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SECTION 1: INTRODUCTION AND LITERATURE REVIEW 



The Transition from Middle School to High School 

Although transitioning from school to school can be challenging at any level, the 
transition to high school may be particularly fraught with challenge because students are 
experieneing a ehange in schools as well as the physieal, emotional, and psyehological changes 
inherent to adoleseence. 

Educators and researchers agree that ninth grade is a pivotal year for students. It is 
typically the year that marks the transition into high sehool and plays a crucial role in setting the 
stage for future edueational outeomes. Success or failure during the freshman year of high sehool 
is directly linked to the probability that a student will drop out before graduation (Legters, 2005). 
Neild, Stoner-Eby, and Eurstenberg (2001) demonstrated that student outcomes in the ninth 
grade eontribute substantially to the researehers’ ability to prediet whether or not a student would 
drop out of high school, over and above student background variables such as family and peer 
relationships. 



What Happens? 

Legters (2005) states that freshmen often comprise the largest class in high schools 
because many of them fail to earn sufficient credits to be promoted to the next grade level. 
Several researchers have deseribed signifieant negative outcomes after the transition into the 
ninth grade, such as a decline in grades and grade point average, decreases in attendance, and 
declining participation in extracurricular activities (Isakson & Jarvis, 1999; Neild & Weiss, 

1999; Reyes, Gillock, Kobus, & Sanehez, 2000; Newman, Myers, Newman, Lohman, & Smith, 
2000). Some students may also develop lower self-esteem and a deereased sense of belonging to 
their school. These outcomes can combine to create feelings of alienation and helplessness in 
students, thereby decreasing their motivation to achieve and increasing the likelihood that they 
may fail and eventually drop out. 

Not all changes are negative. The transition to high school can have positive impacts as 
well. Some students are very excited about becoming high schoolers and look forward to the new 
ehapter in their lives (Mizelle, 2005). Weiss and Bearman (2004) also eontend that the ninth- 
grade transition ean be benefieial for some students, providing a fresh start and allowing students 
to start over in a new school with a clean slate. Most of the literature on the middle school to 
high school transition, however, focuses on the potentially negative effects of the move to ninth 
grade. 



Reasons the Transition May Be So Difficult 

Educators and researchers have offered many thoughts and theories about why the 
transition to high school may be so challenging for new freshmen. High schools are often very 
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different from middle schools in terms of size, social structure, and academic environment. In the 
ninth grade, students typically face a larger school with more students and more teachers and 
also more diversity (Kerr, 2002). The larger school size often equates to a more impersonal 
environment with greater anonymity for students who have less personal contact with their 
teachers (Blyth, Simmons, & Carlton-Ford, 1983; Mizelle, 2005; Roderick, 1993). Because the 
transition generally involves moving to a new building, new ninth graders also have to learn to 
navigate through unfamiliar surroundings. Transitioning to a new high school also inherently 
involves acclimation to a new social environment and structure and becoming acquainted with 
new peers (Mizelle, 2005). Eighth-grade students become accustomed to being the oldest, most 
experienced students in the school; as freshmen, however, they are the youngest and most 
inexperienced students. Some students may even experience what Newman and colleagues 
(2000) call “role loss,” a phenomenon where students are no longer the best athlete or smartest 
student. The sudden changes in social status and role can be stressful for ninth graders. These 
environmental and social changes can often exacerbate feelings of isolation or anonymity and 
inadvertently encourage disengagement and decreased sense of belonging to the school 
(Simmons & Blyth, 1987; Blyth, Simmons, & Carlton-Ford, 1983; Roderick, 1993; Kerr, 2002). 

The academic environment in high schools is also very different from the environment 
students experience in middle school. High schools are more competitive and comparative, and 
grades become much more important (Eccles, Midgley, & Adler, 1984). High schools are more 
likely than middle schools to use ability levels to assign students to tracks, and there is typically 
a greater degree of departmentalization (Roderick, 1993, 1995). Teachers expect more from their 
students and may assign more homework than students had assigned to them in middle school. 
High school curricula are more challenging than middle school curricula, and students are given 
much more responsibility for making a variety of choices that will likely impact their future in 
important ways (e.g., course selection, extracurricular activities) (Mizelle, 2005). 

Despite the greater academic challenges ninth graders face, they are more likely than 
upperclass students to have inexperienced or uncertified teachers (Neild & Farley, 2005). Thus, 
freshmen do not have the benefit of experienced teachers to guide them successfully though the 
transition. Additionally, many freshmen need instruction in basic skills (e.g., time management, 
study skills). Many high school teachers, however, are not prepared or able to teach such skills 
and may take a “sink or swim” attitude to student success. Combining these issues with the fact 
that some students do not have access to various family resources and social support can create a 
difficult transition environment for new freshmen. 
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SECTION 2: CAPITAL HIGH ACADEMY FOR NINTH GRADERS EXCEEDING 

STANDARDS (CHANGES) 



Description of the Academy Initiative 



In the spring of 2004, administrators and school leaders at Capital High School (CHS) in 
Kanawha County, West Virginia, began planning for a small learning community (SLC) within 
the school to help new freshmen successfully navigate the middle school to high school 
transition. CHS administrators decided to undertake the effort after reviewing literature and 
examining data suggesting that new freshmen often struggle to do well during their transition to 
high school. They were concerned that CHS freshmen were struggling, as evidenced by the fact 
that during the 2002-2003 school year, 42% of CHS’s 288 ninth graders were reading below the 
50* percentile, according to standardized test scores. Funding for the SLC effort was secured 
from Kanawha County School District (KCSD) for implementation beginning in the fall 
semester of 2004. 

The new initiative was called Capital High Academy for Ninth Graders Exceeding 
Standards, or CHANGES for short. CHANGES was a school-within-a-school and incorporated 
special scheduling, embedded study skills, research-based instructional and classroom 
management strategies, and culturally responsive teaching and schooling practices. Details are 
provided below. The central aim of CHANGES was to accelerate student achievement and 
increase the pass rate of ninth graders. The goals of CHANGES were to (1) increase the pass rate 
for ninth-grade students and courses at CHS, (2) increase the number of low-achieving students 
enrolling in both honors and/or Advanced Placement (AP) courses during their junior and senior 
years, and (3) increase the number of students with baseline test scores in the 35* to 49* 
percentile range who may select the Professional Pathway. The primary goal (the first listed 
above) was the only goal to focus on performance during the students’ freshman year; the others 
are long-term goals for student success throughout their high school careers. 

During the summer and early weeks of the fall semester of 2004, CHS faculty and staff 
members participating in the CHANGES initiative selected new freshmen to become CHANGES 
(alternately called the “Academy”) students. Eor the most part, students were selected based on 
their performance on the West Virginia Educational Standards Test (WESTEST) during their 
eighth-grade year. Students whose scores fell between the 35* and 49* percentile were eligible 
for selection, although those percentile boundaries were broadened in order to reach the desired 
goal of selecting 60-75 Academy students. Other criteria, such as grade point average (GPA), 
may also have been used to select students into the Academy and reach the desired population 
goal. During the selection process, efforts were made to ensure that the student population of the 
Academy reflected the ethnic diversity of the broader student population of CHS. Parents of the 
CHANGES students were sent letters informing them that their children had been selected to 
participate in this new effort. Parents did have the opportunity to withdraw their students from 
the Academy at any time, although very few elected to do so. Ultimately, approximately 70 
students were enrolled in the Academy, although some of those students were removed from the 
program for various reasons (e.g., transfers, parental choice, suspension). 
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Academy faculty were selected from among the eurrent faculty at CHS because of their 
previous performance at CHS and on the basis of the following factors: a strong desire to work 
with challenging students, willingness to partieipate in professional development, and 
demonstrated knowledge of and competeney in culturally relevant instructional strategies. The 
selected faculty members had a broad range of experience, having served as teaehers from 2 to 
35 years; their length of service at CHS also ranged greatly from 2 to 15 years. During the 
summer of 2004, Academy faculty received special professional development to enhance and 
refine their skills related to such topics as culturally responsive instructional strategies, assertive 
discipline, classroom management, and various teehnology-related skills. School district officials 
felt that these teachers should have extra support for learning to use instructional strategies that 
might be especially helpful with the population of students that would comprise the Academy. In 
general, one faculty member attended the training and disseminated the training to other 
Academy teaehers. The teachers arranged for common planning time throughout the school year 
so they could continue to engage in professional development and planning activities. Academy 
teaehers were also allowed to purchase a limited number of additional student materials (e.g., 
manipulatives, whiteboards) to aid instruction. Academy faeulty and staff attempted to 
eommunicate and work directly with parents to inform them of Aeademy activities and to 
encourage them to influence their children to stay in the Academy and stay foeused on 
academics. 

The Academy was structured as a school-within-a-school. The initial structure of 
CHANGES included an administrator, a counselor, five teaehers, and another teaeher who was to 
provide support and professional guidance to CHANGES classroom teachers. However, early in 
the sehool year, the Aeademy administrator left the school for administrative reasons, and 
Aeademy administrative duties were assumed by the teacher who was initially to provide support 
and guidance. This teacher did not teach an Academy class but did participate in the planning of 
the initiative; she assumed administrative duties in addition to her regular teaching 
responsibilities at CHS. Midway through the year, a retired vice principal was assigned to the 
Academy to help perform some of the administrative and discipline-related functions. The vice 
principal worked only part time (i.e., three days a week) because of state regulations limiting the 
number of days retired educators can work in school systems. Eive CHS faculty members taught 
Academy classes in five subjects: Algebra/Geometry Prep, Coordinated and Thematic Science - 
Nine (CATS9), history, English, and reading. Additionally, study skills were embedded in the 
Academy currieulum and in the form of special instruction during homeroom. All CHANGES 
classrooms were loeated in one wing of the high sehool. The eounselor who participated in the 
CHANGES initiative worked in an office located in the area of the sehool set aside for 
CHANGES classrooms. 

The CHANGES initiative incorporated a special bell schedule different from that used at 
CHS. In the high school, classes were 49 minutes long, with a 20-minute eurrent events 
minielass/homeroom period and a 10-minute break scheduled in the middle of the morning. In 
the Academy, students attended alternating 30-minute classes and 70-minute lab periods, with a 
10-minute break and a 20-minute homeroom period seheduled in the morning. During fifth, 
sixth, and seventh periods. Academy students attended classes in the regular high school 
population. They returned to CHANGES classes for eighth period, which was designed to be an 
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elective class and/or Intro to Majors. Initially, the homeroom period was scheduled at the end of 
the morning before lunch. Based on their experiences in the first semester, however, 
administrators and faculty decided to align the homeroom period during the second semester 
more closely with that of the regular school population thinking that doing so would lead to 
improvements in behaviors and attitudes of the Academy students. Therefore, homeroom was 
moved to mid-morning and was followed by the 10-minute break. The second semester bell 
schedule for the Academy is presented in Table 1, and the bell schedule for CHS in general is 
presented in Table 2. Movement from class to class in the Academy depended on teachers’ and 
students’ ability to abide by the established schedule; there were no bells to mark the beginning 
or end of class periods. 



Table 1: CHANGES Bell Schedule (Second Semester) 



Period 


Type 


Time 


Minutes 


1'’' Period 


Lab 


7:35-8:45 


70 


2"^ Period 


Class 


8:50-9:20 


30 


Homeroom 


Homeroom 


9:25-9:45 


20 


Break 


Break 


9:45 - 10:00 


15 


3'^“ Period 


Lab 


10:00-11:10 


70 


4'*’ Period 


Class 


11:15-11:45 


30 


Lunch A 


Lunch 


11:45-12:20 


35 


5'*’ Period 


Class 


11:51 - 12:40 


49 


Lunch B 


Lunch 


12:40-1:15 


35 


Period 


Class 


12:26-1:15 


49 


7“’ Period 


Class 


1:21-2:10 


49 


8'*’ Period 


Team Elective 


2:16-3:05 


49 



Table 2: Capital High School Regular Bell Schedule 



Period 


Type 


Time 


Minutes 


Period 


Class 


7:35-8:24 


49 


2 ’^“ Period 


Class 


8:30-9:19 


49 


Current Events 


Miniclass 


9:25-9:45 


20 


Break 


Break 


9:45 - 10:01 


16 


3^‘" Period 


Class 


10:01 - 10:50 


49 


4**^ Period 


Class 


10:56-11:45 


49 


Lunch A 


Lunch 


11:45-12:20 


35 


5“ Period 


Class 


11:51 - 12:40 


49 


Lunch B 


Lunch 


12:40-1:15 


35 


6**’ Period 


Class 


12:26-1:15 


49 


7‘^ Period 


Class 


1:21-2:10 


49 


8**’ Period 


Class 


2:16-3:05 


49 
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Another feature of the Academy schedule was alternating A-schedule and B-schedule 
days. The order in which students attended classes varied according to whether the day was 
designated as an “A” day or a “B” day. On an “A” day, students might attend their 30-minute 
history class first, then attend their 70-minute science lab; on the following day (a “B” day), 
students would attend their 30-minute science class during first period and then attend their 70- 
minute history lab session during second period. Typically, Wednesdays were exempt from 
designation as “A” or “B” days because clubs were convened on Wednesdays. Academy students 
followed the regular school bell schedule on those days. Academy students also followed the 
regular school bell schedule on other days designated by the school as Staff Meeting Days 
(SMDs). See Table 3 for the CHS SMD bell schedule. 



Table 3: Capital High School Staff Meeting Day Schedule 



Period 


Type 


Time 


Minutes 


Movement Bell 


Bell 


8:15 


na 


Homeroom 


Homeroom 


8:22-8:42 


20 


1st period 


Class 


8:48-9:31 


43 


2nd period 


Class 


9:37-10:20 


43 


3rd period 


Class 


10:26-11:09 


43 


4th period 


Class 


11:15-11:58 


43 


Lunch A 


Lunch 


12:53-1:27 


34 


5th period 


Class 


12:04-12:47 


43 


Lunch B 


Lunch 


12:04-12:38 


34 


6th period 


Class 


12:44-1:27 


43 


7th period 


Class 


1:33-2:16 


43 


8th period 


Class 


2:22-3:05 


43 



Evaluating the CHANGES Program 



Purpose of the Study 

KCSD and CHS administrators decided to collaborate with an independent third party, 
the regional educational laboratory at Edvantia, Inc. (hereafter, the “lab” or “laboratory”), to 
study the implementation of CHANGES. CHANGES and CHS officials set for themselves the 
goals of determining what effect, if any, participation in the Academy had on students’ course 
passage rates, GPAs, and standardized test scores. Laboratory staff, therefore, examined other 
aspects of the Academy initiative. The focus of study was limited to the implementation of 
CHANGES; questions of impact or effectiveness were not addressed by the study described in 
this report. 

Eirst, because the specific classroom structures, activities, and practices were not clearly 
defined by CHANGES and CHS officials at the start of the 2004-2005 school year, laboratory 
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evaluators proposed to document the strategies, processes, and activities used in the Academy. In 
order for other findings to be meaningful, the instructional strategies, classroom management 
practices, and so on must be documented and described and the intervention defined. The second 
goal of the evaluation was to determine what differences in instruction, if any, exist between 
Academy faculty and regular faculty at Capital High School. This evaluation objective was 
important because Academy faculty intended to employ instructional strategies that differed 
from those typically used by CHS faculty. Finally, because CHS administrators and faculty were 
to examine academic student outcomes, lab evaluators chose to examine student outcomes 
related to perceptions and attitudes. Thus, the third goal of the evaluation was to determine what 
effect, if any, participation in the Academy had on students’ perceptions, attitudes, and skills. 

Audience for this report. The primary audience for this evaluation report consists of 
CHS and KCSD administrators and personnel who are charged with overseeing and 
administering programs like CHANGES. Further, CHS faculty and administrators who 
participated as CHANGES staff during the 2004-2005 school year, or those who may participate 
in CHANGES or similar initiatives in the future, are an important audience for this report. The 
results reported herein may be useful for informing and improving future implementations of 
ninth-grade academies at Capital High. Because this project was funded through the Appalachia 
Education Laboratory at Edvantia, another key audience consists of the Institute of Education 
Sciences (lES) and other U.S. Department of Education officials who support regional 
educational laboratory work. 



Evaluation Questions and Design 

The following primary evaluation questions guided the effort: 

• What instructional strategies and processes are employed in Academy classrooms? 

• What differences in instructional strategies, format, and substance exist between the 
practices of Academy faculty and regular faculty? 

• What are the effects of participation in the Academy on students’ perceptions of the 
academic climate of the school; perceptions of their own academic efficacy; attitudes 
toward school, teachers, and self; perceptions of familial academic support and 
nurturance; and acquisition of various skills? 

These questions provided the structure for the lab evaluation of aspects of the Academy. 

In addition, they allowed project staff to examine issues both of process and outcome. 
Administrators at Capital High School selected three indicators of success for the primary goal of 
the Academy: course passage rates, students’ GPAs, and standardized test scores. CHS staff and 
administrators collected data relevant to these indicators. Laboratory staff agreed to collaborate 
with school staff and administrators to analyze and interpret success indicator data for Academy 
and control students. At the time this report was drafted, CHS staff had not requested assistance 
from lab evaluators in examining or interpreting student academic outcomes. 

Evaluation design. The evaluation incorporated a quasi-experimental design that 
included two components: one focusing on Academy and control group students and one 
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focusing on Academy and regular classroom teaching strategies. The student-focused component 
was designed to incorporate a pre/post design in which target students were to be compared with 
a control group. However, because CHS and Edvantia staff had tremendous difficulty collecting 
signed parental consent forms from Academy and control group students, the design was 
modified (see Appendix C for parental consent forms). The final student-focused evaluation 
design was a posttest only comparison of Academy and control students. As a result of the 
change in the evaluation design and due to CHS officials’ decision to administer a standardized 
achievement test (discussed below), the final evaluation question was modified: 

• What differences, if any, exist between Academy and eontrol students’ perceptions of 
the academic climate of the school; perceptions of their own academic efficacy; 
attitudes toward school, teachers, and self; perceptions of familial academic support 
and nurturance; and perceptions of leadership qualities? 

The modification was made because the original question sought to examine attitudes and 
perceptions with pre/post analyses. The design change made those analyses impossible. Thus, the 
posttest only design relied on the control students to serve as the counterfactual. 

Because Academy students were selected for participation by CHS staff, Edvantia staff 
selected a group of students, matched as closely as possible on relevant characteristics, to serve 
as the control group. Academy and control students were surveyed using various Edvantia 
instruments at the end of their ninth-grade year at Capital High School. Initially, the evaluation 
design called for students to complete three paper- and-pencil instruments: the Measure of 
Academic Supportiveness and Climate (MASC), which assesses students’ perceptions of their 
school and family support for academic endeavors; the Student Attitudes toward Self and School 
(SASS) instrument, which assesses students’ attitudes; and the Student Skills Inventory (SSI), 
which measures students’ acquisition of various academic skills. During negotiations with CHS 
officials, however, Edvantia staff learned that CHANGES faculty planned to administer the 
Terra Nova, a standardized achievement test. Because collection of SSI data would have 
duplicated the data collection efforts of CHS, Edvantia evaluation staff decided to eliminate that 
instrument from the design and thereby reduce the paperwork and response burden of the student 
participants. Thus, the final student design can be depicted in the following manner: 

2004-2005 School Year Spring 2005 
Academy Students X Oa Ob 

Control Students Oa Ob 



Oa = MASC Ob = SASS 
X = Participation in the Academy 

The teaching-focused component consisted of systematic classroom observations. 
Because Academy faeulty received some of their professional development during the summer 
before the start of school and because the Academy was implemented at the start of the 2004- 
2005 school year, Edvantia researchers were unable to collect any “pure” pre-treatment data 
regarding classroom strategies used by faculty. Classroom observation data were collected and 
reviewed throughout the year to assess instruction by Academy and regular faeulty (i.e., non- 
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Academy faculty who served as the teacher control group). This time series design can be 
depicted as 



Academy Teachers 


Summer 2004 

XpD 


Fall 2004 

Oi Xa 


Winter 2004/2005 

O2 Xa 


Spring 2005 
O3 


Control Teachers 




Oi 


O2 


O3 



XpD = Professional Development Xa = Implementation of the Academy 

O = Classroom Observations 



Additionally, Academy faculty completed implementation logs to describe strategies, activities, 
and events employed in the Academy classrooms. These implementation logs were reviewed to 
provide a clearer understanding of the processes of the Academy and to aid in the interpretation 
of quantitative findings. To add richness and depth to the findings of the evaluation and to 
further explore the context and implementation of the Academy, in-depth interviews were 
conducted with CHANGES faculty during the summer of 2005. 
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SECTION 3: EVALUATION METHODS 



Samples 

The evaluation design and evaluation goals ealled for comparisons between the Academy 
and regular CHS classrooms, both in terms of student attitudinal outcomes and in terms of 
teaching strategies and classroom processes. Therefore, Edvantia staff devised a plan for 
selecting a sample of non-Academy ninth-grade students, matched as closely as possible to 
Academy ninth-grade students on relevant variables, to serve as a control group. In order to 
compare teaching strategies, classrooms in which control group students were enrolled were 
selected as control classrooms. 



Student Samples 

All students selected by the administration at CHS to participate in the Academy 
composed the treatment group. As mentioned previously, new freshmen students were selected 
to participate in the Academy on the basis of their eighth-grade achievement test scores, which 
were mostly in the lowest two quartiles, and grade point averages. Lab evaluators received the 
first list of Academy participants in mid-September 2004 and received the final list of students 
selected into the Academy in early October 2004. Approximately 70 students were initially 
enrolled in the Academy; by the end of the 2004-2005 school year, however. Academy 
enrollment was 61 students. Lab evaluators received signed parental consent forms for 56 
Academy students whose parents agreed that they could participate in the study. 

A sample of students was selected to serve as a control group. In collaboration with the 
counselors and administrators at CHS and KCSD, Edvantia evaluators received a list of all new 
freshmen students (by student identification number) not selected for participation in the 
Academy. Through a matching process, control group students were selected based on how 
closely they resembled Academy participants on the basis of ethnicity, gender, middle school 
attended, and middle school achievement test scores. Every attempt was made to match control 
students to Academy students as closely as possible with regard to achievement test scores in 
order to reduce variance between the groups. The process was iterative and required three rounds 
of matching. There were problems with the initial matched sample submitted to CHANGES staff 
in late September 2004. It included the ID numbers of students who had been selected into the 
Academy after the initial list of participants had been received by lab evaluators; also, some of 
the students listed were not enrolled at CHS. The second sample included similar issues. The 
final sample of control students’ ID numbers was submitted to CHANGES staff in early October 
2004, and the names of those students were returned to Edvantia staff in mid-November 2004. It 
was not possible in all cases to achieve an exact match; the resulting control sample tended to 
have slightly higher levels of performance on the WESTEST. However, other characteristics 
(e.g., gender, ethnicity) were similar across the two student samples. Only 15 consent forms were 
received from the 65 students who ultimately were selected into the control sample. Researchers 
acknowledge that those students whose parents gave consent for participation in the control 




11 



group (n = 15) may have differed from their counterparts whose parents did not give consent for 
them to participate in the study. 

Because there were an unequal number of students in the Academy and control groups 
who were eligible to participate in the data collection (i.e., had signed parental consent), 
evaluators selected a subsample of CHANGES students for comparative analyses. Evaluators 
wanted to achieve samples that matched as closely as possible in terms of student demographic 
characteristics. Therefore, when comparative analyses on the student instruments were 
performed, data for CHANGES students who shared similar characteristics with the control 
students were used. Data for other Academy students not selected into the subsample were not 
included in the analyses. 



Classroom Samples 

Academy teachers were selected to serve in that capacity by the administration at CHS. 
All five of the teachers were women and White, and each one taught a different core curriculum 
subject (mentioned previously). All five teachers’ classrooms served as the treatment sample for 
the examination of strategies, processes, and activities used in the Academy classrooms. In order 
to select a control sample of classrooms, evaluators drew on the sample of control students. The 
evaluators, in coordination with CHS staff, collected the class rosters for the core curriculum 
classes in which control group students were enrolled. Erom these rosters, the evaluators selected 
one control classroom per subject (e.g., English, science). This method of control classroom 
selection was employed to enable the comparison of Academy teaching strategies with the 
teaching strategies being used to instruct the control group students. Eive control classrooms, one 
per subject area, were selected to serve as the control classroom sample for classroom 
observations and instructional comparisons. 



Instruments 



Implementation Logs 

The laboratory evaluators collaborated with Academy faculty to gather data about 
treatment implementation and Academy activities. Prior to the start of the 2004-2005 school 
year, the evaluators consulted with Academy administrators and teachers to design a structured 
activity log to be used by Academy teachers for documenting events, activities, instructional 
strategies, discipline issues, and other happenings in Academy classrooms. Logs were designed 
for weekly entries and included a reflective component that allowed teachers to reflect on which 
strategies worked best, which strategies did not work, and possible alternative strategies for the 
future. Likewise, because the Academy initiative employed a counselor to work specifically with 
CHANGES students, laboratory staff collaborated with the counselor to design a counselor’s log. 
The weekly counselor’s log included documentation of the number of students counseled, the 
number of parents contacted, reasons students sought or were referred for counseling, and 
referrals made by the counselor. The implementation logs were designed collaboratively so that 
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they would be useful both to evaluators and to Academy staff who wanted to use the logs as part 
of professional development discussions during planning and professional development sessions. 
These implementation logs were reviewed to provide a clearer understanding of the processes of 
the Academy and to aid in the interpretation of quantitative findings. See Appendixes A and B 
for copies of the teachers’ implementation log and the counselor’s implementation log, 
respectively. Control group teachers did not keep logs; however, intensive classroom 
observations provided systematic evidence of the classroom behaviors of both groups of 
teachers. 



Special Strategies Observation System-Revised 

Evaluators used the Special Strategies Observation System-Revised (SSOS-R) to collect 
systematic classroom behavior data for Academy and control classrooms three times during the 
2004-2005 school year. This system, composed of four instruments, is designed for use in a 
variety of settings to systematically collect data on essential elements of classroom behaviors 
related to instruction, management, and context. A unique feature of the SSOS-R is that it can be 
employed to collect classroom data on a teacher; a specific targeted student; or both during an 
observation period, as well as to provide snapshots of the entire classroom. The SSOS-R is a 
viable instrument for school effectiveness research due to its strong grounding in the current 
literature on effective teaching and its utilization of a variety of methodologies. This combination 
of instruments generates low-, moderate-, and high-inference data; this triangulation of data 
sources further documents the veracity of the data collected. The four instruments include the 
Classroom Observation Form, QAIT Assessment of Classroom, Standards Performance 
Continuum, and Classroom Environment and Resources Checklist. The entire observation could 
last a maximum of 60 minutes. Figure 1 portrays the SSOS-R system as a clock, with the time 
allotments specified for each instrument. The four instruments are described more fully below. 

Classroom Observation Form (COF). The COE is a combination observation system 
that is best described as a category system with low-inference items and multiple coding 
procedures (Nesselrodt & Schaffer, 1993; Sullivan & Meehan, 1983). It is based on the Stallings 
Observation System (Stallings, 1980) and the Classroom Activity Record designed by Evertson 
and Burry (1989). The top page of the COF collects typical demographic information, including 
observer, date, teacher observed, number of adults and students in the class, grade level, ethnicity 
and gender of subjects being observed (teacher and target student), and type of class (Academy 
or control). 

The COF segment of the observation includes a maximum of 58 minutes — 2 minutes for 
coding the cover page and then 56 minutes for coding classroom behaviors. The 56 minutes are 
divided into seven 8-minute time periods; each 8-minute block is captured on a separate page. 
The first minute of each block focuses on the entire classroom and provides a class snapshot by 
looking at both student engagement (the number of students on task, off task, waiting, or out of 
the room) and groups and activities (whether students are clustered in teacher, aide, or student 
groups and their type of involvement, such as working alone, management, interaction, or 
socialization). The remainder of each 8-minute block (either 7 or 8 minutes, depending on the 
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length of time required to fill in the snapshot information) is devoted to observing either the 
teacher or “target” student. 



For the CHANGES project, researchers decided to include both the teacher observation 
and a target student. The focus of the observation switched from teacher to target student for 
each 8-minute block. There were a total of 28 discrete, mutually exclusive activities that could be 
chosen to describe the teacher and target student behaviors. 
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Figure 1: 

Graphic Depiction of the Special Strategies Observation System-Revised (SSOS-R) 

QAIT Assessment of Classroom. This instrument is best described as a moderate- 
inference, simple coding, rating device. QAIT stands for Quality of Instruction, Appropriate 
Level of Instruction, Incentives for Learning, and Use of Time. This two-page instrument 
contains 40 items grouped under those four major categories. Each item has a Likert-type 
response scale of 1 to 5 {unlike this class to like this class). This instrument was completed at the 
end of each observation session. 
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Standards Performance Continuum (SPC). This instrument is a rubric used to quantify 
the implementation of the Standards for Effective Pedagogy (Tharp, Estrada, Dalton, & 
Yamauchi, 2000). It is best described as a high-inference, simple coding, rating device. The SPC 
contains five standards labeled Joint Productive Activity, Language and Literacy Development, 
Contextualization, Challenging Activities, and Instructional Conversation. Each standard is rated 
on a Likert-type response scale of 0 to 4 (not observed to integrating). This instrument was 
completed at the end of each observation. 

Classroom Environment and Resources Checklist (CERC). This instrument is a low- 
inference, simple coding, sign system. This one-page checklist contains 16 classroom attributes 
that are coded either as present or not present, such as adequate lighting, use of multiracial 
materials, posted assignments, etc. Next, 18 classroom resource items, such as textbooks, 
computers, and worksheets, are listed. Observers indicate first whether such resources were 
visible or not. If visible, observers then indicate whether the resources were used during the 
observation. This instrument was to be completed at the end of each observation session. 

The SSOS-R instruments were originally tested and used in a pilot test of the evaluation 
for the Kentucky Extended School Services program (Nesselrodt & Schaffer, 2000a, 2000b). 
They were then modified and converted to a scannable format by Edvantia staff in 2001 and used 
in the evaluation of the Kentucky Extended School Services program (Cowley et ah, 2002) and 
again in the evaluation of a culturally responsive teaching pilot project in Kanawha County 
schools during the 2003-2004 school year (Hughes et al., 2004). Thus, these instruments possess 
face and content validity and have proven their utility in prior research. Eurther, a high degree of 
inter-rater reliability was achieved among the data collectors during the 2001 and 2003 SSOS-R 
training sessions. 

In a previous project using the SSOS-R instrument (see Hughes et al., 2004), Cronbach’s 
alpha coefficients were computed to assess the degree of internal consistency reliability. The 
reliabilities for that administration of the SSOS-R were moderate to high. Eor the COE 
instrument, the activity code section coefficient was .51, the student engagement coefficient was 
.76, and the grouping strategy coefficients were .76 for the grouping section and .38 for the 
number of students section. For the QAIT instrument, the coefficient for all 40 items was .96; by 
scale, the coefficients were .95 for quality of instruction, .69 for appropriate level of instruction, 
.91 for incentives for learning, and .91 for use of time. For the CERC instrument, the coefficient 
for all 50 items was .87; by section, the coefficients were .78 for the environment items, .81 for 
the visible resource items, and .44 for the used resource items. 



Measure of Academic Supportiveness and Climate (MASC) 

The Measure of Academic Supportiveness and Climate (MASC) was administered to 
Academy and control students in May 2005. The 42-item instrument assesses students’ 
perceptions of themselves as students and of their school experiences and also asks students 
about their families’ awareness of and involvement in their children’s school lives. Respondents 
rate each item using a scale ranging from 1 to 5 (not at all true to always true). The four scales 
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within the instrument are: Student Belonging, Family Expectations, Student Academic Efficacy, 
and Eamily/School/Student Involvement. Taken together, these subscales assess the degree to 
which students think that their schools and families provide them with academic nurturance and 
support, and the extent to which students view themselves as intellectually capable. In previous 
administrations, the MASC has demonstrated satisfactory reliability, with Cronbach’s alphas 
from .78 to .93 for the four subscales and .95 for the overall instrument (Cowley, Riffle, Howley, 
Voelkel, & Ermolov, 2004). Cowley and colleagues also found that the MASC possesses 
satisfactory concurrent validity with the Miami-Dade School Climate Survey (r = .56, p < .01). 



Student Attitudes Toward Self and School (SASS) 

The Student Attitudes toward Self and School (SASS) was administered to Academy and 
control students in May 2005. The 39-item instrument assesses students’ attitudes (e.g., self- 
efficacy, enjoyment of school, perceptions of teacher supportiveness). Respondents rate each 
item using a scale ranging from 1 to 5 (strongly disagree to strongly agree), and the overall 
instrument has demonstrated satisfactory reliability in previous administrations (a = .92). 
Although the original authors of the SASS instrument (University of Maine, 1999) proposed 
eight scales (Belonging, Heroes, Sense of Accomplishment, Eun and Excitement, Spirit of 
Adventure, Curiosity and Creativity, Leadership and Responsibility, and Confidence to Take 
Action), Wilson, Wilson, Cowley, Meehan, and O'Keefe (2003/2004) found that with a large 
sample of West Virginia students, four scales emerged, two of which represent an internal locus 
of control where the other two represent an external locus of control: 

• Self-Efficacy: describes a student’s ability to identify a problem and take corrective 
action 

• Teacher-Centric: focuses on student’s perceptions of the teacher's attitudes toward 
the student and the student's belief about their teacher’s supportiveness 

• Leadership: focuses on effective leadership and includes being excited about seeking 
the solution to problems; the scale further reflects the influence of both peers and 
teachers on leadership perceptions 

• Like School: describes a student’s ownership of learning and the resulting enjoyment 
Taken together, these subscales assess conditions that support high levels of aspirations in youth. 



Faculty Interviews 

A semi-structured interview protocol was developed by an Edvantia evaluator and a 
consulting research fellow (see Appendix D for a copy of the interview protocol). The interviews 
primarily focused on the four major program components described in the CHANGES proposal: 
professional development, curriculum and instruction, the alternative/block schedule, and 
parental involvement or parent-teacher contact. Additionally, to a small degree, the interviews 
explored the nature and prevalence of challenges faced by teachers and administrators on the 
CHANGES faculty. Interviews were designed to last approximately 30 minutes and were semi- 
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structured to allow additional probing and exploration of responses in order to discover the most 
pertinent and useful information. 



Data Collection 



Implementation Logs 

The CHANGES teachers and counselor were instructed to complete their implementation logs 
each week in order to capture the most accurate data about the instructional strategies, classroom 
practices, and activities or events that occurred each week. The logs were provided to Academy 
staff members in electronic form, for convenience and ease of completion and data entry. Three 
teachers completed their implementation logs and submitted them electronically each week. Two 
other teachers and the counselor, however, felt more comfortable completing their 
implementation logs in a paper- and-pencil format. The evaluators were amenable to this 
adjustment in data collection. The paper- and-pencil implementation logs were collected by the 
CHANGES support teacher (who assumed the duties of the original Academy administrator) and 
submitted to Edvantia evaluators throughout the year. 



Systematic Classroom Observations 

The six data collectors who were involved in the CHANGES classroom observations 
were formally trained in the SSOS-R system in the fall of 2004, prior to any observation data 
collection. Eive of the data collectors were assigned to specific classrooms (one Academy 
classroom and one control classroom each), and one data collector was chosen as an alternate in 
case of scheduling conflicts. All primary data collectors were able to complete the observations; 
the alternate was not required to perform any classroom observations for the CHANGES project. 

Observations took place during the 2004-2005 school year at three time periods: 
December 2004, late Eebruary and early March 2005, and May 2005. Each data collector 
conducted two approximately hour-long observations of each Academy classroom and each 
control classroom for each of the three time periods. Laboratory evaluators observed Academy 
classrooms during the 70-minute lab sessions and during the 30-minute class sessions. Eor each 
observation time period, the observers completed one visit to a 70-minute lab session and two 
visits to the same 30-minute class (being mindful of the alternating A/B day schedule). 

Observers made two visits to the 30-minute class sessions to complete the full 56-minute COE 
observation. The two classroom visits were counted as one complete observation. Thus, a total of 
30 Academy classroom observations (15 lab session and 15 class session) and 30 control 
classroom observations were completed during the 2004-2005 school year. Each data collector 
completed her classroom observations individually; however, the observation team was 
consulted if observers had questions about how to code particular classroom events or activities. 

All data collectors used the SSOS-R forms to record data systematically during the 
classroom observations. The COE instrument was fully completed during the classroom 
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observation. Because the Academy lab sessions were 70 minutes in length, Edvantia observers 
completed the 56-minute COF and were able to complete most items for the QAIT, the CERC, 
and the SPC during class time. However, it was not always possible for the data collectors to 
fully complete the QAIT, SPC, and CERC while in the classroom because some observations ran 
consecutively or because class ended before observers could complete the forms. These 
instruments were completed as soon after the observations as possible. 

SSOS-R COF observations allow for data to be collected about teacher and/or student 
behaviors. In this project, Edvantia observers collected data both for the teacher and for a target 
student in each classroom observed. For the Academy classrooms, any student could be chosen 
as the target student, given their participation in the program. For the control classrooms, each 
data collector was given a class list identifying those students who had been selected into the 
control sample of students. It was then up to the individual data collector to choose a student 
from the list to serve as the target student for that observation. Up to four of the eight-minute 
COF segments could be coded for teacher behavior, and up to three of the segments could be 
coded for target student behavior. 



Student Instruments 

The MASC and the SASS were administered to Academy and control students whose 
parents had given consent for them to participate in the evaluation project. Among Academy 
students, 56 students had signed parental consent forms on file; among control students, Edvantia 
had records indicating parental consent for 15 students. Administration of these two instruments 
occurred in May 2005, with all students being provided sufficient information about the project 
to give their assent to participate. Because Academy teachers were administering the Terra Nova 
to CHANGES students at that time, the evaluators and CHANGES staff agreed that the 
Academy teachers would administer the instruments to the 56 students with parental consent at 
the end of Terra Nova testing. The lab evaluators prepared instrument administration packets and 
provided them to each of the teachers. The packets included administration instructions for the 
teacher (see Appendix E), student assent information (see Appendix F), the MASC, the SASS, 
and a large manila envelope in which students were to seal their completed questionnaires. 
Academy teachers then collected the envelopes containing the completed questionnaires and 
turned them in to a central collection point (i.e., the Academy counselor’s office), where an 
Edvantia evaluator retrieved them. 

Because control students were distributed in many classes throughout the regular CHS 
ninth-grade population, a special data collection session was held to administer the MASC and 
the SASS. Students whose parents had signed consent forms were called out of class at the 
beginning of one class period and asked to report to a classroom. There, an evaluator 
administered the two student instruments using the same administration packets that had been 
prepared for the Academy teachers. Completed questionnaires were collected in the sealed 
envelopes, and the control students were free to return to their classes as soon as they had 
submitted their questionnaires. Students required approximately 20 minutes to complete both 
paper- and-pencil questionnaires. 
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Faculty Interviews 

Interviews with CHANGES faculty members occurred during the late summer of 2005. 
The interviewer was able to contact and interview five of seven faculty members either in person 
or via telephone. In-person interviews were conducted either at Capital High School or at 
Edvantia’s office in Charleston, West Virginia. Eaculty members who were not available to 
arrange an in-person interview were contacted by telephone. Interviews required approximately 
30 minutes to complete. 



Analyses 

All data collected throughout the CHANGES project were entered into electronic 
databases (e.g., SPSS, Excel) and cleaned. Quantitative data were analyzed with descriptive and 
inferential statistical tests, as appropriate for the type of data and research question. Qualitative 
data, such as responses to implementation logs, were analyzed by question or topic, as 
appropriate. Data was segmented into passages through coding, and themes were identified and 
given broad codes. Einer coding was applied as necessary, using patterns emerging within each 
broad category of responses. Themes were then tabulated to provide a general, quantitative 
analysis of the most salient and prevalent issues. The following section presents more specific 
information for analyses employed for each data collection instrument. 



Implementation Logs 

Teacher and counselor implementation log data were entered into Excel files. Each 
question was assigned a separate sheet within the Excel workbooks. Responses to the first two 
items, which were included at the request of Academy teachers and administrators, were not 
analyzed further because their purpose was to provide immediate information to Academy staff 
for discussions during common planning time throughout the school year. Quantitative data, such 
as the number of parent contacts and effectiveness ratings for instructional strategies, were 
analyzed via descriptive statistics (e.g., frequencies, means). Qualitative data, such as reflections, 
were coded by question according to common themes. 



Classroom Observations 

Because the SSOS-R is a machine-scannable instrument, laboratory staff designed data 
entry templates using Remark scanning software. SSOS-R data were scanned by observation 
period; data files were then cleaned and exported to SPSS. Eollowing data cleaning, all three data 
files were merged into one master file for statistical analysis. 

COF. COE activity data provided the number of minutes spent in any of 28 discrete 
activities for both the teacher and the target student. These numbers were summed across the 
eight-minute intervals for each observation by both teacher and target student. Up to four of the 
eight-minute blocks focused on the teacher and three on the target student (if the observation 
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lasted a full hour). These data were analyzed using the 28 individual activities and by collapsing 
the data into four main categories of teacher-led, student-led, management/organization, and off- 
task. 



COF classroom snapshot data provided information on student engagement (i.e., the 
number of students on task, off task, out of the room, or waiting during the first minute of each 
eight-minute block) and on grouping and activities (i.e., the number of students involved with the 
teacher, any aide, or other student groupings, along with the type of activity taking place). The 
student engagement numbers were summed across the eight-minute intervals for each 
observation by classroom and also by determining the percentage of students engaged in each 
category (on task, off task, out of room, waiting). These data were analyzed using the four 
engagement codes of on or off task, out of the room, and waiting. For the groups and activities 
segment, the number of students involved with the teacher, aide, or other students by activity 
(interactive, working alone, management, or social/uninvolved) were summed across the eight- 
minute intervals for each observation. These data were analyzed by number of students per 
activity. Further, one-way ANOVAs were conducted for all of the COF data (activity, student 
engagement, and groups and activities) to determine if statistically significant differences existed 
among the Academy and control groups. Effect sizes were computed as appropriate. 

QAIT. QAIT data were analyzed by creating four scales composed of the 40 individual 
items: quality of instruction, appropriate level of instruction, incentives for learning, and use of 
time. Since there were unequal numbers of items in each scale, the item scores were summed and 
then averaged to generate the scale score. Descriptive statistics were used to describe results for 
the Academy and control groups. Further, one-way ANOVAs were conducted to determine if 
statistically significant differences existed among the group scale scores. As appropriate for the 
analyses, effect sizes were also computed. 

CERC. CERC data were analyzed by calculating frequency percentages showing 
whether the classroom attributes were present and whether the classroom resources were visible 
and used during the observations of Academy and control classrooms. 

SPC. Data for each of the five SPC items were analyzed by calculating means for both 
the Academy and control groups. Means and standard deviations were calculated per SPC item, 
and independent t tests were computed for each item mean. The alpha level for these t tests was 
set at the .05 level. Effect sizes were computed, as appropriate. 



Student Instruments 

To compare the differences on the MASC and on the SASS between Academy students 
and control students, independent t tests were computed on each subscale mean for each 
instrument. The alpha level for these t tests was set at the .05 level, and effect sizes were 
computed, as appropriate. 
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Faculty Interviews 

Faculty interviews were audio-recorded, and transcripts and notes were produced. 
Interview transcripts and notes were coded and summarized according to general descriptive 
categories. Pattern coding (Fetterman, 1989; Yin, 2003) was used to discover patterns among 
individuals and descriptive categories. Patterns were searched for opposing or inconsistent data. 
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SECTION 4: FINDINGS FROM CHANGES IMPLEMENTATION 



The following section presents the findings of analyses conducted on the various data 
collected throughout the CHANGES evaluation. Findings are presented by data collection 
instrument. Conclusions about the implementation of the CHANGES initiative are presented in 
the next section. 



Teacher Implementation Logs 



Academy teachers were asked to complete weekly implementation logs to provide 
quantitative and qualitative data regarding the Academy classes and activities. Over the course of 
the school year, teachers submitted a total of 172 implementation logs. Teachers’ responses to 
the eight implementation log items analyzed for the evaluation are detailed. 



Parent Contacts 

Academy teachers made 541 contacts or attempted contacts with Academy students’ 
parents or guardians during the school year. Contacts were in the form of telephone calls, parent- 
teacher conferences, e-mails, notes, and letters. Contacts were made to update parents on student 
progress, inform parents of behavioral problems, and provide parents with positive comments 
regarding student performance and/or improvement. Table 4 presents the number of parent 
contacts made by each teacher (designated by a letter), the total number of contacts, and the 
mean number of contacts per week. 



Table 4: Parent Contacts made by CHANGES Teachers 



Teacher 


Number of Logs 


Number of Contacts 


Mean per Week 


A 


36 


95 


2.79 


B 


35 


163 


4.66 


C 


29 


49 


1.69 


D 


35 


141 


4.15 


E 


37 


93 


2.51 


Total 


172 


541 


3.15 



Instructional Strategies 

Academy teachers were asked to indicate which instructional strategies they used during 
the week and to rate the effectiveness of each strategy using a scale of 1 {not at all effective) to 5 
{extremely effective). The ratings for each strategy were summed and divided by the number of 
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responses to each strategy to create a mean score ranging from 1 to 5. Table 5 provides 
descriptive statistical summaries for each strategy. 

The most frequently used instructional strategy, according to implementation log 
responses, was “set goals or objectives for lessons,” which was indicated in 168 of the 172 logs 
submitted (97.7%). The second most frequently cited strategies were “lessons and/or instruction 
were related to state learning standards” and “questioned or cued students to check for 
understanding,” which were indicated in 165 (95.9%) and 164 (95.4%) logs, respectively. The 
least-used instructional strategy was “generated or tested hypotheses,” which was indicated in 
only 53 logs. The most effective instructional strategy for Academy teachers was “active 
listening to student responses,” which had a mean effectiveness rating of 4.75, very close to a 
rating of extremely effective. The least effective strategy was “homework and practice,” which 
was indicated on 1 10 implementation logs and, with a mean rating of 3.71, was the only strategy 
to receive a mean rating of less than 4.00. 



Table 5: Implementation Log Descriptive Statistics for 
Effectiveness of Instructional Strategies 



Strategy 


N 


% 


Mean 


SD 


Set goals or objectives for lessons 


168 


97.67 


4.28 


0.75 


Lessons and/or instruction were related to state learning standards 


165 


95.93 


4.68 


0.52 


Questioned or cued students to check for understanding 


164 


95.35 


4.52 


0.71 


Reinforced and/or recognized student effort 


162 


94.19 


4.52 


0.7 


Gave information and/or illustrated concepts with a variety of 
approaches/strategies 


158 


91.86 


4.33 


0.64 


Active listening to student responses 


157 


91.28 


4.75 


0.54 


Provided feedback to students about progress toward objectives 


155 


90.12 


4.5 


0.71 


Connected new information to prior knowledge 


153 


88.95 


4.13 


0.83 


Graphic organizers or non-linguistic representation 


136 


79.07 


4.56 


0.59 


Scaffolds instruction 


116 


67.44 


4.02 


0.92 


Learning groups/cooperative learning 


114 


66.28 


4.17 


0.89 


Summarizing or note taking 


112 


65.12 


4.04 


0.87 


Homework and practice 


110 


63.95 


3.71 


0.97 


Asked students to compare, contrast, classify, or use analogies/metaphors 


109 


63.37 


4.21 


0.77 


Generated or tested hypotheses 


53 


30.81 


4.41 


0.92 



Comments About Instructional Strategies 

The implementation log included an item asking Academy teachers to comment about the 
instructional strategies they used during the week. The purpose of the item was to probe more 
deeply into teachers’ thoughts about the use and effectiveness of the various strategies. Some 
teachers included lengthy comments describing their classroom activities, and others provided 
shorter thoughts. In all, 252 comments were offered in response to this item. Edvantia evaluators 
categorized the comments by theme, and Table 6 presents the number of comments in each 
category, accompanied by representative quotes from the implementation logs. The teachers 
commented extensively about specific activities employed in the classroom, although they did 
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not always mention specific instructional strategies. They also commented a great deal on 
students’ enthusiasm about, resistance to, or difficulties with the activities. 



Table 6: Teacher Comments regarding Instructional Strategies 
Used During the 2004-2005 School Year 



Category 


N 


Representative Comments 


Classroom activities (description) 


63 


• I gave them study guides for the test. 

• To review, we played a relay game working in four teams to 
answer questions related to the concepts covered in the unit. 

• Monday, we reviewed one last time for test taking questions and 
having students provide explanations. 

• This week was mainly a week for reviewing. 

• This week, we finished making our projects and then we had our 
competition. 


Students doing well or working hard 


46 


• The students continued to work diligently on their projects. 

• We can see students scaffolding information on their own. 

• They worked well in groups together and were very efficient in 
turning in work this week. 

• They worked very well together and independently, taking a lot 
of pride in their work and completing it on time. 

• Students continued to succeed with the kinesthetic activity. . . 

• Most students worked very hard on their study guides. 


Difficulties 


32 


• The students are having problems with nonlinguistic 
representations. 

• Difficult to individually instruct students, as others easily get off 
task. 

• Students were not eager to do this [work]. They would say, “Why 
do I have to do this. I’m going to the NBA” or they would 
research other interests. 

• They were . . . very distracted by the upcoming holiday this week. 
They were not very receptive to using their time wisely to finish 
necessary work and activities before the break. 


Students engaged or enjoy activities 


32 


• For the most part, all of the students are very involved and 
excited. They enjoyed being able to create something. 

• They really enjoy using [white boards] to complete problems. 

• They enjoyed [an activity] and were teaching one another about 
the proper steps in solving equations. 

• They are very excited about creating and completing their . . . 
projects. 

• They love graphic organizers and work in pairs. 


Effective strategies 


18 


• The students did an excellent job with the Marzano vocabulary 
this week. They made great nonlinguistic representations and 
they did help them remember the words. 

• As we worked the process step by step, I kept changing the 
markers on the board as well as on the transparency on the 
overhead when they took notes and copied several examples. 
They said it helped them recognize the changes in steps. 

• Writing to learn has been a great help because it allows me to 
come back the next day and review if needed. 

• The most effective strategies so far are the use of graphic 
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Category 


N 


Representative Comments 






organizers and comparison/contrast activities that make visual 
connections for the students and/or connect prior knowledge to 
current content. 


Teacher excitement 


10 


• The students really impressed me this week. 

• This was so exciting to see that the implementation of this 
strategy really does work! 

• Students are really catching on . . . that is exciting. 


Technology use 


9 


• The students were very excited about being able to use the 
computers. 

• The internet activity that I did this week worked well with a few 
classes and not so well with others. 


Students improving 


6 


• Students are continually showing improvement in the cooperative 
learning setting. 

• They are beginning to feel safe enough to answer questions out 
loud. 


Suggestions for changing activities 


6 


• They need to choose the material. . .trip to the library next time! 

• This would have gone much smoother as an independent activity. 


Time-related concerns 


5 


• Labs are taking a long time to complete. 


Grades 


4 


• Test results followed a general bell curve of mostly C’s and D’s 
with a few A’s but still too may F’s. 

• Scores were excellent on the post-test. 


Student concerns 


3 


• Getting ready for the final - students are scared ! 


Productive work time 


2 


• This week was very productive. 


Expectations of students 


2 


• Students are finally realizing that we are serious about grades and 
that they must be earned not received. 


Other 


14 


• I was out for three days this week because of an illness, so I tried 
to plan activities that were very student centered. 

• Students were not as attentive this week. They were very hyper 
and with our new no tolerance policy on tardiness, they did make 
it on time to class, but complained a lot about the new policy. 

• ... students had another alternative schedule because of a pep 
assembly at the end of the day. This once again cut classes short 
and made it difficult for students to do any work. 

• Students seem ready for the test. 



Classroom Procedures and Classroom Management Strategies 

Academy teachers reported whether any of six specific classroom procedures or 
classroom management strategies were used during the week. Teachers reported the use of 
clearly stated goals, objectives, and expectations most frequently {n = 165, 95.9%), with slightly 
less use of posted rules, schedules, procedures, and so on in their classrooms {n = 162, 94.2%). 
The least-used classroom procedure was displaying student work or homework in Academy 
classrooms (n = 84). Table 7 presents the frequency with which each strategy was employed, 
based on the 172 implementation logs that teachers submitted. 
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Table 7: Frequency of Use of Classroom Procedures and Management 
Strategies in CHANGES Classrooms 



Procedure or Strategy 


N 


% 


Clearly stated goals, objectives, and expectations 


165 


95.93 


Rules, schedule, procedures, etc. posted in classroom 


162 


94.19 


Built positive teacher- student relationships 


151 


87.79 


Disciplinary interventions 


109 


63.37 


Visible or posted rubrics and/or scoring guides 


101 


58.72 


Displayed student work/homework in classroom 


96 


55.81 


Other 


1 


0.58 



Teachers were provided an opportunity to list other procedures or management strategies not 
included in the original list of six; only one other classroom procedure was mentioned (“daily 
sponge activity and word of the day”). However, that particular comment may more 
appropriately be called a classroom activity (see the following section). 



Classroom Activities/Events 

The implementation logs asked Academy teachers to indicate the occurrence of six 
specific activities or events in their classroom each week. Recognition for achievement was the 
most frequently cited event, being indicated in 146 of the 172 logs (84.9%). Discipline issues (n 
= 121, 70.4%) were also a frequent occurrence in Academy classrooms. All other activities and 
events were indicated in fewer than 100 of the 172 weeks worth of implementation log data. 
Table 8 presents the frequencies with which each activity was indicated on the teachers’ 
implementation logs. 



Table 8: Frequency of Events and Activities in CHANGES Classrooms 



Event or Activity 


N 


% 


Achievement recognitions 


146 


84.88 


Discipline issues 


121 


70.35 


Student presentation(s) 


83 


48.26 


Technology time or activities 


67 


38.95 


Speaker or outside presentation 


23 


13.37 


Lab(s) 


17 


9.88 


Other 


9 


5.23 



The implementation logs provided an opportunity for teachers to indicate whether or not any 
other classroom activities or events occurred during the week. In addition to the activity listed in 
response to the classroom procedures item, nine other classroom activities were listed by 
Academy teachers. Teachers listed such activities as tests and quizzes (n = 2), nonlinguistic 
representations, work on class projects (n = 2), PowerPoint presentations, review games, and 
field trips (n = 2). 
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Events Potentially Impacting Students 

Teachers were asked to provide information about events that occurred in the school that 
might have impacted students’ ability to focus, learn, or interact appropriately with staff and 
other students. Teachers offered 245 thoughts in response to this item. Edvantia evaluators 
categorized the comments according to themes; Table 9 presents the number of comments in 
each category as well as representative quotes from the implementation logs. Most of the 
teachers’ comments indicated that disruptions of the regular schedule (e.g., delays, major events 
in the school, assemblies) were the most frequent events that impacted students’ concentration, 
behavior, and ability to focus on academic endeavors. 



Table 9: Teachers’ Comments Regarding Events That Potentially Impacted 
CHANGES Students During the 2004-2005 School Year 



Event Category 


N 


Representative Comment 


School Closings, Delays, or Altered 
Bell Schedule 


58 


• Monday - 2 hr. delay due to 3 degree temperature. 

• Students had an abbreviated week due to ISE day, Friday. 

• Wednesday was a Club Day. 

• Every Wednesday, we have some type of activity in the 
morning. This will always cause interruptions in the schedule. 

• ... Staff Meeting Day on Wednesday, which makes for shorter 
class periods during the day as a result of school starting an hour 
later than usual. This is a difficult day because our students have 
a tendency to respond to abrupt changes in the routine with 
defiant and very hyperactive behavior. 


Major School or Academy Events, 
Assemblies 


50 


• One pep assembly on Friday caused a lack of concentration 
during the final period of the day in anticipation of what the 
assembly would be like. 

• This was homecoming week, need I say more. 

• Behavior modification field trip to the Clay Center - students 
anxious, excited about going, disappointed about not going. 

• The only difficult day was Friday because of the Prom Fashion 
Show. 

• All ninth-grade students had to view a video by Dr. Phil on 
bullying . . . the kids were not very receptive to it and made fun 
of it as a whole. 

• Thursday field trip to Pittsburgh. 


Holidays 


23 


• Election Day 

• No School Veteran’s Day. 

• This is three days before holiday break - many students did not 
attend school on Wednesday. 

• The students were very excited about Christmas break. 


Substitute Teachers 


19 


• Two teachers were out for a day or two. 

• I was . . . out with strep throat and the students had a substitute 
who was not qualified to teach them. 

• I was out of the building, and subs reported that behavior was 
not good in my classes. 


Report Cards and Progress Reports 


13 


• The students received their progress reports from the entire 
school on Friday. 

• Grade Report 
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Event Category 


N 


Representative Comment 


Report Cards and Progress Reports 
(con’t) 




• Report cards on Thursday definitely had an effect on students’ 
behavior and attitude on academics. 


Discipline Related 


11 


• Bells held for one class due to an altercation in the office (Code 
Blue situation). 

• Also, a fight did occur among students within our CHANGES 
group ... It was handled quickly. 

• Several students have big issues with each other and it is truly 
creating a disturbance. 


Testing/Exams 


9 


• Altered exam schedule allowing for 90 minute exams and 
shorter review periods. 

• Students were not happy having to take the Terra Nova test 
again. 


School Timelines 


9 


• I believe the students’ behavior is deteriorating because it is 
close to the end of school and some students feel the year is 
already done. 

• Perhaps they were anxious over the end of the first nine weeks 
on Thursday. 

• They had to get adjusted to being back to school after the break. 


Weather 


9 


• The week was unexpectedly shortened due to inclement 
weather. 

• Rainy weather - behavior Wednesday. 


Location Changes 


6 


• Our students were displaced from the regular classroom for two 
days so that our classrooms could be used for testing. 

• WESTEST - we were moved to different classrooms. 


Student Schedule Changes 
(permanent) 


6 


• All of the students’ schedules were changed, so some students 
were upset because they were no longer with their friends. 

• An abrupt change was made in our daily schedule to follow the 
school’s regular bell schedule to eliminate tardies and help 
provide a transition for students to their sophomore year. 


General Comments 


4 


• It was a very smooth week overall. 


Other (including community) 


28 


• Students also have dealt with a murder in their neighborhood 
.... This devastated some students and definitely affected the 
climate of the school because the murder victim has a daughter 
that attends Capital High. 

• Tuesday - a student in the program moved to California. 

• Having two new students join our group which got the students’ 
wound up again. 



Reflections 

Lastly, teachers were asked to provide reflections in the form of ideas, insights, 
suggestions, or changes. The purpose of the item was to elicit feedback about and suggestions for 
the CHANGES initiative, including teachers’ thoughts and reflections about Academy structures 
and processes. The directions on the implementation log, however, were not explicit, and many 
of the comments in response to this item were general reflections or comments about activities, 
events, strategies, or other factors not specifically related to the CHANGES initiative. Of the 126 
reflections offered in response to this item, 102 concerned general reflections about classroom 
activities, school events, or student behaviors that were not specifically related to the structures, 
functions, or processes of the Academy. Twenty-four comments, however, were more directly 
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related to Academy functions overall. Most of these reflections took the form of suggestions for 
changing or improving CHANGES. Table 10 presents representative comments. 



Table 10: Teachers’ Reflective Comments 



Category 


N 


Representative Comment 


Related to CHANGES 


24 


• ... next time we take a field trip, the right to go should not be a 
behavior modification contingency for students. The students that were 
left behind were the very students that needed to have the opportunity 
to go and see what the Clay Center had to offer . . . behavior 
modification is a definite need with our program, yet it should not 
prohibit students from going on our special field trips .... 

• Open House was positive for parents . . . who attended. I believe we are 
all on the same train. 

• Moratorium on changes to program. 

• The students need a lot of positive feedback. Next year, we might not 
want to do the same activities throughout the first week of school. The 
kids were complaining about doing the same things in some of the 
classes and being bored. 

• My only concern about the field trip was that many students were left 
behind. They did not qualify for the trip. I hope that the next trip we 
have will include everyone. Many of the students left behind, in my 
opinion, are the kids we want to give experience to. They do not have 
people in their lives who take them to different places. 

• We need to find subs who are reliable and know what they are doing. 

• I really like the changes which have been made to the students' 
schedules. 

• I think every Wednesday should be a school bell schedule when Club 
Day starts. This would make all Wednesdays the same for our students. 
They seem to appreciate being like the rest of the student body some 
times, but enjoy extra activities other times. This way there would also 
be some consistency throughout the week and Wednesdays would have 
approximately 43 minute classes. 


Other (not specifically related 
to CHANGES) 


102 


• Teaching them is getting easier, but there are still a few who haven’t 
caught on yet. 

• They were confident about their computer skills after 2 days ! 

• It was a pleasant week. 

• This unit was way too long. It would be more effective broken down 
into much smaller units and goals. 

• I’m anxious to finish grading the quizzes to check for understanding. 

• This week was very productive. The students had much better behavior 
than they have in the last couple of weeks. 

• I think this unit went very well. It was much shorter than others we 
have done. 

• Students still are struggling with group work because of their constant 
efforts to socialize, as opposed to doing work. Also, students are 
horrible about turning in homework assignments, which are hurting 
their grades tremendously. I am struggling with what to do to resolve 
this issue. 

• The next time, I will give options when we do a project. I only had 
three students who really did not enjoy this project. In order to make 
sure everyone participates, I will give two options. 
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Category 


N 


Representative Comment 


Other (not specifically related 
to CHANGES) (con’t) 




• I thought this week went very smoothly. The students are very eager to 
study and do well on their final in my class. I was also excited by many 
of the answers I got from students. They were able to remember a lot of 
the information we have covered this semester. I was also happy with 
the amount of students who came to me asking for help on their time. 



Other Comments 

Although given an opportunity to provide other comments, Academy teachers seldom 
took that opportunity in completing their implementation logs. Only in 15 of the 172 
implementation logs did teachers offer additional comments. The comments were general 
comments about the mood of the week, miscellaneous activities not previously addressed in the 
log, thanking other teachers for assistance, or relaying student thoughts or reactions. Of note 
were two particular comments about student reactions. In one comment, a teacher stated that 
“students will make an adjustment to CHANGES. [They] need reassurance that we are not 
dumbing down the curriculum, but they must work to achieve success.” In another comment a 
teacher indicated that multiple instructors in one classroom “creates some confusion,” and that 
“students have expressed dissatisfaction” with the arrangement. 



Counselor Implementation Logs 



Like the teachers in the Academy, the CHANGES counselor was asked to keep weekly 
implementation logs to document the number of students counseled, the reasons the students 
sought or were referred to counseling, referrals made for follow-up, the number of parents 
contacted, and so on (see Appendix B for a copy of the counselor implementation log). The 
CHANGES counselor submitted 35 logs over the course of the 2004-2005 school year, including 
one for a week during which she was absent and one for Spring Break week. Those two weeks 
were not included in tallies and averages; thus, the number of logs analyzed for this report was 
33. The following section presents the information recorded on the counselor’s implementation 
logs. 



Number of Students Counseled 

Over the course of the 2004-2005 school year, the CHANGES counselor had 1,159 
meetings with students. Some students met with the counselor more than one time during the 
year. The number of students counseled each week ranged from a low of 0 to a high of 84. On 
average, about 35 Academy students were counseled each week. 
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Number of Parents Contacted 

In all, the CHANGES counselor contacted parents 167 times throughout the school year. 
Contacts took the form of phone calls, conferences, or, in some cases, home visits. The number 
of parents contacted each week ranged from 0 to 15. The counselor contacted an average of 5 
parents per week during the year. 



Reasons Students Sought Counseling 

The CHANGES counselor was asked to indicate why students sought or were referred to 
counseling each week, using a list of eight possible reasons. Students were most often in 
counseling for academic reasons (n = 29; indicated on 29 of the 33 logs) and home or family 
problems or concerns (n = 28). Students were least often counseled for psychological needs or 
teacher referrals (n = 6 each). Interestingly, counseling because students had conflicts with 
school staff (n = 16) tended to cluster around the end of each semester. Table 1 1 presents the 
frequencies for each counseling category. 



Table 11: Frequency of Occurrence of Each Reason Students Sought Counseling 



Reason 


N 




Academic Counseling 


29 


87.88 


Home or Eamily Problems 


28 


84.85 


Social Problems or Issues Outside of School 


26 


78.79 


Conflicts or Issues with School Staff 


16 


48.48 


Career Counseling 


15 


45.45 


Conflicts or Issues with Other Students 


15 


45.45 


Psychological Needs 


6 


18.18 


Teacher Referrals 


6 


18.18 



* Percent of 33 logs. 



The counselor log included an opportunity for the counselor to list other reasons students 
sought counseling. However, the counselor did not list any other reasons in response to that item. 



Referrals Made 

Because counselors make referrals to other school staff members, school programs, 
community programs, social workers, or other entities or officials, the implementation log asked 
the CHANGES counselor to indicate the individuals, programs, or other entities to which she 
referred students (or that she called in reference to students) throughout the year. Referrals were 
most often made to social workers or case managers (n = 26, indicated on 26 of the 33 logs) and 
to the school nurse (n = 25). No students were referred to Upward Bound throughout the school 
year. Referrals were made to three other individuals or entities not included in the list of seven 
specified on the implementation log. These referrals were to a consultant working with 
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CHANGES staff, the school principal, and In-School Suspension (ISS). Table 12 presents the 
number of weeks during which referrals were made to each of the various entities. 



Table 12: Number of Referrals Made to Other Entities or Programs 



Entity or Program 


N 




Social Workers or Case Managers 


26 


78.79 


School Nurse 


25 


75.76 


Law Enforcement 


7 


21.21 


Student Assistance Team 


6 


18.18 


Mental Health or Psychological Services 


4 


12.12 


Social Services or DCFS* 


4 


12.12 


Other 


3 


9.09 


Upward Bound 


0 


— 



* Percent of 33 logs. 

** DCFS = Department of Children and Family Services 



Referrals to the school nurse and to social workers were evenly distributed throughout the 
school year. All referrals to the student assistance team, psychological services, and “other” 
entities were made during the first semester. Four of the seven referrals to law enforcement 
occurred in the weeks around the end of the first semester and beginning of the second semester. 



Events Potentially Impacting Students 

The counselor was asked to provide information about events or occurrences in the 
school that might have impacted students. The purpose of this question, like the identical 
question on the teachers’ implementation log, was to discover any events that might disturb 
students’ ability to focus on academic work. Of the 40 events noted by the counselor, the two 
most frequently cited categories were major school or Academy events (e.g., homecoming, 
assemblies, field trips; n = 10) and school delays or closures {n = 8). Most of the comments 
related to disruptions in the school or class schedules. Six comments, however, mentioned report 
cards or progress reports, and two other logs cited testing or exams as events that affected 
students. Table 13 presents the number of comments in each category and some representative 
quotes. 



Table 13: Counselor Comments Regarding Events That Potentially Impacted 
CHANGES Students During the 2004-2005 School Year 



Event Category 


N 


Representative Comment 


Major School or Academy Events 


12 


• Homecoming Week 

• Friday - Prom Fashion Show 

• CHANGES took students to Pittsburgh for the day 5 am - 
10:30 pm. Wonderful experience for the students. 


School Closings, Delays, or Altered Bell 


8 


• Early morning staff meeting - students start 1 hr. late. 
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Event Category 


N 


Representative Comment 


Schedule 




• 2 hr delay due to snow 


Report Cards and Progress Reports 


6 


• Progress reports were distributed for the first time. 
Students were in an uproar concerning their grades. 


Holidays 


3 


• Thanksgiving Break 


Testing/Exams 


2 


• Final Exam Week 


School Timelines 


2 


• End of T' nine weeks 


Discipline Related 


2 


• Severe behavior problems with some students. They are 
interfering with the learning of others. 


Other 


2 


• Substitutes in two classes 

• Club Day 


General Comments (not specific to events) 


3 


• ISS - In School Suspension (students refusing to go) 



Consultations 

Included on the counselor’s implementation log was an item asking the counselor to 
describe consultations made during each week. For this item, the CHANGES counselor listed the 
people with whom she consulted. Of the 91 comments, the largest number (n = 20) concerned 
consultations with other faculty or staff members at CHS, excluding administrators. 

Consultations with administrators were noted 12 times, and consultations with parents were 
mentioned 1 1 times. In addition to indicating the roles or positions of people with whom she 
consulted, the counselor also noted that some consultations were about academic counseling (n = 
3), scheduling (n = 3), discipline issues (n = 4), or career counseling (n = 5). Table 14 presents 
the number of comments made in each category and includes representative quotes. 



Table 14: Counselor Comments Regarding Consultations Made for 
CHANGES During the 2004-2005 School Year 



Category 


N 


Representative Comment 


Faculty and Staff 


20 


• Other Counselors 

• Team meetings daily with teachers 

• Teachers during planning 

• Counselors re: scheduling for next year 


Administrators 


12 


• Vice Principal 

• Administrators 


Parents 


11 


• Parents were called in because of severe discipline problems. 

• Parent conferences by phone re: skipping by a few 

• Parents 


Social Worker 


9 


• Social Worker re: referrals 5+ days absent 

• Social Worker 


Other 


9 


• Patchwork 

• County Alternative Placement Committee 

• Mentors for teen in crisis 


Nurse 


8 


• Nurse - ill students (asthma, hurt leg) 

• School nurse 


Probation Officer 


5 


• Probation Officer 


Career Counseling 


5 


• Career counseling with [teacher’ s] 8* period 

• Classroom career counseling sessions 
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Discipline Counseling 


4 


• Discussions centered around attendance problems. . . 


Academic Counseling 


3 


• Group counseling with students - academic failure 


Scheduling 


3 


• Students re: 2"'^ semester schedules 


Consultant 


2 


• Consultant 



Reflections 

As in the teachers’ implementation log, the counselor log included a reflective 
component intended to capture the counselor’s impressions of the Academy effort, suggestions 
for improvement, changes needed in Academy structure or functioning, and so on. For the most 
part, comments offered in response to the “reflections” item were general comments not 
necessarily related specifically to the Academy. Seventeen of the 29 comments were reflections 
about specific events, student attitudes, and so on. However, 12 comments did relate, more or 
less, to the structures, functions, or activities of the Academy. Most of these reflections noted 
difficulties, successes, or suggestions for future implementations. For instance, the counselor 
reflected that students needed to be more organized, that some students felt like they were “in a 
hole” with their grades, that parents were being supportive of the Academy effort, and that 
CHANGES students should be encouraged to participate in more activities. Table 15 presents 
representative comments from each category of responses. 



Table 15: Counselor Reflective Comments 



Category 


N 


Representative Comment 


Related to CHANGES 


12 


• Students appear to need to be organized. They have many problems 
because they don’t complete homework. Many have not returned 
contracts or AEL slips. 

• We need to have a unit on social skills. 

• Next year, CHANGES students need to be encouraged to participate 
more in activities. 

• A behavior management plan is needed for the team. Students feel that 
rules are not enforced the same for all; some students do their ISS, and 
other complain and are not made to do it. Assistant principal of 
CHANGES is working on scheduling; time on discipline is not here. 


Other (not specifically related 
to CHANGES) 


17 


• The first week of school went smoothly. 

• More acting out this week for some - Holiday season brings stress in 
teens. 

• Change of weather - Students ill with colds, flu. 

• Students are expressing negative attitudes. Their behavior says they think 
school is over. 

• Discipline problems continue. 



Other Comments 

The last item on the log provided an opportunity for the counselor to offer any other 
comments about the week. Sixteen comments were included on 11 logs. Most of the comments 
(n = 6) mentioned that various students had left the CHANGES program for various reasons 
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(e.g., transfers, suspensions). One comment mentioned that a new student joined the group. 
Other comments (n = 3) were personal in nature (e.g., “I was off [work] two days.”), and some 
(n = 5) related to other work the counselor had done that week such as scheduling work, 
covering a class for another teacher, completing paperwork, and making a home visit. One 
additional comment mentioned that a student was sent home because of discipline problems. 



SSOS-R Classroom Observations 



A total of 60 classroom observations were completed during the 2004-2005 school year. 
For each teacher selected, data collectors completed 2 observations at three different time periods 
over the year (December 2004, February/March 2005, and May 2005), for a total of 6 
observations per teacher. 

There was a maximum of 56 minutes possible for the actual coding of classroom 
behaviors (seven 8-minute COF segments). The average number of minutes of classroom coding 
per observation was 50.7. The median number of minutes was 49, and the mode was 48. The 
classroom coding ranged from a low of 42 minutes (2%) to a high of the maximum 56 minutes 
(20%). A total of 3,039 classroom behavior minutes were coded. Adding 4 other minutes per 
observation (for initial cover sheet coding, QAIT, CERC, and SPC) and multiplying by 60 
observations (240 minutes) brings the total number of observation minutes to 3,279 minutes or 
55 hours (equivalent to roughly 7 full days of observations). 

The number of adults (teachers or aides) in the classrooms ranged from 1 to 3; the 
average number was 1 for both Academy and control groups. The number of students per 
classroom ranged from 6 to 23; the average number was 1 1 for Academy classrooms (SD = 3) 
and 16 for control classrooms (SD = 4). Other demographic information for both groups is 
presented in Table 16. Of note are the observations that neither the Academy nor the control 
group included African American teachers, and the Academy group included no male teachers. 
Also, more African American students were selected as target students in Academy classrooms 
(n = 20), and more White students were selected as target students in control classrooms (n = 

24). 



Table 16: Demographic Information by Group for Classroom Observations 



Demographic Variable 


Grouping 


Academy 


Control 


N 


% 


N 


% 


Teacher ethnicity: 


African American 


0 


0% 


0 


0% 


White 


30 


100% 


30 


100% 


Teacher gender: 


Female 


30 


100% 


23 


77% 


Male 


0 


0% 


7 


23% 
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Demographic Variable 


Grouping 


Academy 


Control 


N 


% 


N 


% 


Target student ethnicity: 


African American 


20 


67% 


5 


17% 


Biracial/Multiracial 


2 


7% 


1 


3% 


White 


8 


27% 


24 


80% 


Target student gender: 


Female 


8 


27% 


23 


73% 


Male 


22 


73% 


7 


27% 


Subjects: 


English 


4 


13% 


10 


33% 


History 


6 


20% 


6 


20% 


Mathematics 


6 


20% 


6 


20% 


Reading 


8 


27% 


2 


7% 


Science 


6 


20% 


6 


20% 



The SSOS-R provides a “elassroom snapshot” that looks at whole-class student 
engagement and grouping configurations every 8 minutes and “ongoing activities” of the teacher 
and a target student alternately for 7 or 8 consecutive minutes (depending on whether the 
snapshot coding took an entire minute). There was a maximum of seven 8-minute blocks; four of 
these were focused on the teacher and three were focused on the target student. During these 
blocks, 28 discrete activities could be coded in time segments of 1 to 8 minutes. As noted earlier, 
these blocks equaled a maximum of 56 minutes of the 60-minute observation; 2 minutes were 
dedicated to preliminary coding on the cover page at the beginning of the observation and 2 
minutes were allocated to completing the QAIT, SPC, and CERC at the conclusion of the 
observation. 



Classroom Snapshot: Student Engagement^ 

The four categories within student engagement included number of students on task, off 
task, out of the room, and waiting (e.g., for the next task, for assistance). The average number of 
students on task for the Academy classrooms was 57, with a standard deviation (SD) of 17; for 
the control classrooms, the average was 86, with an SD of 27. For students off task, the average 
for the Academy classrooms was 8 (SD = 5); for control classrooms, the average was 6 (SD = 7). 
For students out of the room, the Academy classroom average was 4 (SD = 4); the control 
classroom average was 2 (SD = 3). Finally, for the number of students waiting, the Academy 
classroom average was 5 (SD = 9); the control classroom average was 5 (SD = 7). 

The percentage of students engaged in each category (on task, off task, out of the room, 
or waiting) was calculated by group (Academy, control). According to Stallings (1980), student 



* Note: Numbers for student engagement may seem high; however, the reader is reminded that observation data are 
collected for seven time segments throughout the class time. Engagement figures are calculated using the summed 
data from all seven segments, thereby causing an appearance of inflated figures. 
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engagement rates above 80% have been associated with high gains in student achievement. The 
percentage of students on task for the control classrooms was above 80%, and for the Academy 
classroom, the percentage was 77%. Table 17 provides descriptive statistics for the student 
engagement variables (on task, off task, out of room, and waiting) by group (Academy, control). 



Table 17: Descriptive Statistics for Number of Students in 
SSOS-R Student Engagement by Group 



Engagement Category 


Grouping 


Academy Classrooms 


Control 


1 Classrooms 


Mean 


SD 


% 


Mean 


SD 


% 


Students On Task 


57 


17 


77% 


86 


27 


87% 


Students Off Task 


8 


5 


11% 


6 


7 


6% 


Students Out of Room 


4 


4 


5% 


2 


3 


2% 


Students Waiting 


5 


9 


7% 


5 


7 


5% 



One-way ANOVAs were computed to determine whether statistically significant 
differences existed within these four variables (on task, off task, out of the room, and waiting) 
between groups (Academy, control). The two categories of on task and out of the room resulted 
in statistically significant differences. With one large effect size at 1.29 and one medium effect 
size at .65, these findings suggest that there was practical significance in favor of the control 
classrooms. This difference, however, may be a function of the greater number of students 
observed in control classrooms. See Table 18 for a summary of the results. 



Table 18: One-Way ANOVA Results for Number of 
Students in SSOS-R Student Engagement 



Engagement Category 


df 


F 


P 


Effect size 


Students On Task 


1,58 


24.78 


.000 


1.29 


Students Off Task 


1,58 


0.99 


.323 


— 


Students Out of Room 


1,58 


4.66 


.035 


0.56 


Students Waiting 


1,58 


0.05 


.828 


— 



Classroom Snapshot: Groups and Activities 

This section of the COF focused on the teacher, aide, or student groups, as well as the 
activity taking place (interaction, working alone, management, or social/uninvolved). Table 19 
provides descriptive statistics for the groups and activities variables (teacher, aide, or student 
groups and interaction, working alone, management, or social/uninvolved activities) by group 
(Academy, control). A few points are worth noting: no teachers in either group were coded as 
working alone or social/uninvolved, neither group had students engaged in management 
activities, and more students were coded as social/uninvolved in control classrooms. 
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Table 19: Descriptive Statistics for Number of Students in 
SSOS-R Groups and Activities by Group 



Variables 


Grouping 


Acad 


lemy 


Control 


Mean 


SD 


Mean 


SD 


Teacher Interaction 


32.33 


16.55 


57.55 


36.73 


Teacher Working Alone 


— 


— 


— 


— 


Teacher Management 


9.67 


13.14 


36.12 


39.81 


Teacher Social/Uninvolved 


— 


— 


— 


— 


Aide Interaction 


52.00 


— 


— 


— 


Aide Working Alone 


— 


— 


— 


— 


Aide Management 


0.33 


0.58 


— 


— 


Aide Social/Uninvolved 


— 


— 


— 


— 


Student Interaction 


16.00 


8.49 


47.00 


— 


Student Working Alone 


32.24 


15.54 


43.93 


30.40 


Student Management 


— 


— 


— 


— 


Student Social/Uninvolved 


13.62 


10.85 


15.47 


8.38 



One-way ANOVAs were generated to determine whether statistically significant 
differences existed within these 12 variables (interaction, working alone, management, or 
social/uninvolved for teacher, aide, and student groups) by group (Academy, control). Two of 
the 12 ANOVAs did identify significant differences between groups. Teacher interaction resulted 
in a significant difference by group: the control classrooms had significantly more students 
interacting with teachers. However, the control classrooms also had significantly more students 
involved in teacher management activities than the Academy classrooms. Both effect sizes were 
large (.89 each), indicating there was practical significance in favor of the control classrooms. 
This difference, however, may be a function of the greater number of students observed in 
control classrooms. See Table 20 for a summary of these results. 



Table 20: One-Way ANOVA Results for Numbers of 
Students in SSOS-R Groups and Activities 



Groups and Activities Category 


df 


F 


P 


Effect size 


Teacher Interaction 


1,44 


9.27 


.004 


0.89 


Teacher Management 


1,21 


5.68 


.027 


0.89 


Student Interaction 


1,1 


8.90 


.206 


— 


Student Working Alone 


1,30 


1.95 


.173 


— 


Student Social/Uninvolved 


1,41 


0.37 


.544 


— 
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Ongoing Activities 

This section of the SSOS-R included two components that alternately focused on the 
teacher (four 8-minute blocks) and a target student (three 8-minute blocks). The first component 
was to indicate which of 28 discrete activities was being observed during that particular time 
period (see Table 21 for a complete listing of all 28 activities, along with a brief description of 
each). The second component was to indicate how many minutes were spent engaged in a 
particular activity. The smallest time increment was 1 minute; the largest was 8 minutes (each 
page was an 8-minute block). The maximum number of teacher-focused activity minutes per 
observation was 32; for student-focused activity minutes, the maximum number was 24. 



Table 21: SSOS-R Individual Activities and Descriptions 



Activity 


Description 


A. Teacher presentation of 
Content 


Presenting academic content to whole class. Includes lecture, demonstration, and 
explanation of academic content. May include questioning or comments from 
students, but mainly to inform students, introduce/explain materials, including 
previously introduced material. 


B. Recitation or discussion 


Providing students practice of skills/review of materials. Includes questioning of 
students, short written tasks, or content-oriented game/board work, interactive 
review, or reviewing textbook exercises. 


C. Directions for 
assignments 


Teacher is explaining to class exact procedures for doing an assignment, seatwork 
activity, or homework. Can include headings, numbering, or any information about 
form in which the assignment is to be done. 


D. Small-group instruction 


Teacher works with group of two or more students. 


E. Tests 


Students work independently on a test, quiz, readiness test, or assessment; teacher 
may read questions aloud, as in a spelling test. 


F. Checking 


The teacher and students are going over seatwork problems, a quiz, or assignment 
for the purpose of checking/grading it in class. Little/no teacher explanation or 
review is entailed. 


G. Procedural or behavioral 
presentation 


Presents/reviews class/school rules/procedures. Should be used when 
instituting/explaining class procedures/rules governing student behavior or when 
giving class feedback on behavior, discussing problems relating to behavior, or 
following class procedures. 


H. Administrative routines 


Teacher or student is checking attendance, making announcements, opening or 
closing routines without academic content, discussing grades, distributing graded 
papers, recording grades, or changing seating. 


I. Transitions 


The teacher and students are involved in activities that entail changing from one 
activity to another, i.e., moving between small groups, getting supplies or materials 
for a different activity, or passing papers. 


J. Nonacademic activity 


Teacher involved with students in activities such as games, discussions, or 
television that are not related to content of the class. 


K. Discipline 


Students are involved in some discipline for misbehavior, i.e., putting their heads 
down on desks for a period of time for misbehavior. 


L. Praising class 


The teacher is praising one or more students for work or tests completed, for 
behavior, etc. 


M. Monitoring 


The teacher is moving around the room giving feedback to individual students or 
groups, or is providing feedback during individual or group student presentations. 


N. Not occupied 


Teacher or students are not engaged in academic learning, neither are they involved 
in any type of nonacademic activity, i.e., just sitting at desk, etc. 
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Activity 


Description 


0. Off task 


Teacher is involved in a nonacademic activity, i.e., talking to another teacher, 
talking on the phone, etc. Students are not doing whatever they were instructed to 
do, i.e., they are “goofing off,” talking to a classmate, “doodling,” etc. 


P. Out of room 


Teacher or students have left the room for some reason, such as going to restroom, 
going to some type of pull-out program, going home sick, etc. 


Q. Individual instruction 


Teacher works with an individual student on content-related material. 


R. Waiting time 


Students have no assigned task. Either they are finished and have no other 
assignment or they are just waiting for the next activity. 


S. Individual seatwork 


Students are working at desks individually. This code includes activities that are 
content-centered. Brief directions for seatwork or short teacher interruptions to 
explain or clarify directions would be left in seatwork time unless they last more 
than one minute. 


T. Individual seatwork at 
computer 


Students are working at computers individually. This code includes activities that 
are content-centered. Brief directions or short teacher interruptions to explain or 
clarify directions would be coded unless they last more than one minute. 


U. Pairs or group seatwork 


Students are involved in content-centered student- or teacher-initiated group 
projects or small-group tasks. 


V. Pairs or group seatwork 
at computer 


Students are grouped in pairs or groups at computers and are performing content- 
centered activities. 


W. Sustained writing or 
composition 


Students are involved in sustained writing. 


X. Sustained reading 


Students are involved in sustained reading. 


Y. Hands-on learning 


Individual students or groups are using manipulatives to enhance learning, including 
experiments. 


Z. Independent inquiry or 
research 


Students are working individually or in groups to conduct research for a unique 
product. 


#. Student-initiated 
questions 


Individual students generate in-depth (higher order) questions for the teacher. 


!. Student presentations 


Students are involved individually or as a group delivering content to the class. 



The 28 activities were grouped into four main categories: teacher led, 
management/organization, student led, and off task. Table 22 provides the classification of each 
activity into one of these four categories. 



Table 22: Classroom Observation Individual Activities by Main Category 



Main Categories 


Individual Activities 


Teacher Led 


A. Teacher presentation of content 

B. Recitation or discussion 

C. Directions for assignments 

D. Small-group instruction 

E. Tests 

F. Checking 

L. Praising class 
Q. Individual instruction 


Management/Organization 


G. Procedural or behavioral presentation 

H. Administrative routines 

I. Transitions 
M. Monitoring 
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Main Categories 


Individual Activities 


Student Led 


S. Individual seatwork 

T. Individual seatwork at computer 

U. Pairs or group seatwork 

V. Pairs or group seatwork at computer 

W. Sustained writing or composition 

X. Sustained reading 

Y. Hands-on learning 

Z. Independent inquiry or research 
#. Student-initiated questions 

! . Student presentations 


Off Task 


J. Teacher nonacademic activity 

K. Discipline 

N. Not occupied 

O. Off task 

P. Out of room 
R. Waiting time 



On a global level, the average number of minutes spent on teacher-led activities when the 
focus was the target student was 1 1.58 with a standard deviation (SD) of 6.31; for the teacher 
focus, the average was 15.02 with an SD of 6.05. The average number of minutes spent on target 
student-focused management/organization activities was 2.50 with an SD of 1.98; for the teacher 
focus, the average was 1 1.73 with an SD of 5.77. For target student-focused, student-led 
activities, the average number of minutes was 8.24 with an SD of 5.13; for the teacher focus, the 
average was 3.00 with an SD of 0.00. For target student-focused, off-task activities, the average 
was 5.82 with an SD of 4.25; for the teacher focus, the average was 2.87 with an SD of 3.47. 
Table 23 provides descriptive statistics for the main activity categories (teacher led, 
management/organization, student led, and off task) by teacher focus or target student focus and 
by group (Academy, control). In general, the teacher-led activities were most prevalent for both 
target student and teacher focus across both groups. 



Table 23: Descriptive Statistics for Mean Number of Minutes in SSOS-R Main Activity 
Categories by Teacher or Target Student Focus by Group 



Activity Category 


Grouping 


Academy 


Control 


Mean 


SD 


Mean 


SD 


Teacher Focus 


Teacher Led 


16.80 


5.60 


13.23 


6.04 


Management/Organization 


13.63 


5.04 


9.76 


5.88 


Student Led 


— 


— 


3.00 


— 


Off Task 


1.75 


0.97 


4.09 


4.72 


Target Student Focus 


Teacher Led 


9.07 


6.05 


14.38 


5.43 


Management/Organization 


2.87 


2.34 


2.13 


1.54 
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Activity Category 


Grouping 


Acad 


lemy 


Control 


Mean 


SD 


Mean 


SD 


Student Led 


8.38 


5.53 


8.10 


4.75 


Off Task 


5.81 


4.29 


5.83 


4.29 



One-way ANOVAs were eomputed to determine whether statistically significant 
differences existed within these eight variables (teacher led, management/organization, student 
led, and off task by either target student or teacher focus) by group (Academy, control). Three of 
the eight ANOVAs did identify significant differences by group. For teacher focus, the 
categories of teacher led and management/organization were statistically significant by group. For 
target student-focus, the category of teacher led resulted in a statistically significant difference by 
group. With two moderate (.61 and .71) and one large (.92) effect sizes, these findings indicate that 
there was practical significance. See Table 24 for a summary of results. 



Table 24: One-Way ANOVA Results for Mean Number of Minutes in SSOS-R 
Main Activity Categories by Teacher or Target Student Focus 



Category 


df 


F 


P 


Effect size 


Favor 


Teacher Focus 


Teacher Led 


1,58 


5.62 


.021 


0.61 


Academy 


Management/Organization 


1,57 


7.40 


.009 


0.71 


Academy 


Student Led 


— 


— 


— 


— 


— 


Off Task 


1,21 


2.83 


.107 


— 


— 


Target Student Focus 


Teacher Led 


1,53 


11.65 


.001 


0.92 


Control 


Management/Organization 


1,30 


1.15 


.292 


— 


— 


Student Led 


1,43 


0.03 


.858 


— 


— 


Off Task 


1,48 


0.00 


.993 


— 


— 



QAIT 



The QAIT assessment of classroom instrument measured four features of the classroom: 
quality of instruction, appropriate level of instruction, incentives for learning, and use of time. 
Forty specific features were rated on a 1 to 5 Likert-type scale (1 = unlike this class to 5 = like 
this class). A QAIT form was completed for each of the 60 classroom observations. 

The Academy group scored highest on the teacher using effective management (mean of 
4.00, SD of 0.83), on necessary time being allocated for instruction (mean of 3.90, SD of 0.80), 
and on teachers using an appropriate pace to cover content (mean of 3.93, SD of 0.91). Both 
groups scored lowest on the teacher using academic incentives such as small groups with 
individual incentives (mean of 1.23 each, 5Z)s of 0.43 and 0.63, respectively). Table 25 provides 
descriptive statistical information for each QAIT item by group (Academy, control). 






42 



Table 25: Descriptive Statistics of SSOS-R QAIT Items by Group 



QAIT Items 


Grouping 


Academy 


Control 


Mean 


SD 


Mean 


SD 


Quality of Instruction 


1. Lesson makes sense to students. The teacher: 




la. Organizes information in an orderly way. 


3.83 


0.91 


3.80 


1.00 


lb. Notes transitions to new topics. 


3.33 


1.30 


3.27 


1.34 


Ic. Uses many vivid images and examples. 


3.43 


1.22 


2.57 


1.36 


Id. Frequently restates essential principles. 


3.73 


0.94 


3.67 


1.16 


2. Lessons relate to students’ background. The teacher: 




2a. Uses devices such as advanced organizers. 


2.70 


1.32 


1.90 


1.03 


2b. Reminds students of previously learned materials. 


4.03 


0.96 


3.63 


1.13 


3. The teacher exhibits enthusiasm. 


3.97 


0.96 


3.37 


0.96 


4. The teacher shows a sense of humor. 


3.03 


1.19 


2.73 


1.41 


5. Lesson objectives are clearly specified. The teacher: 




5a. States lesson objectives orally or in writing. 


3.17 


1.12 


2.93 


1.11 


5b. Conducts formal and/or informal assessment. 


3.87 


1.11 


3.87 


0.78 


5c. Provides immediate and corrective feedback. 


4.37 


0.72 


4.07 


0.83 


6. Teachers use an appropriate pace to cover content. 


3.93 


0.91 


3.30 


1.06 


Appropriate Level of Instruction 


7. Instructional strategies match students’ abilities. The teacher: 




7a. Accommodates students’ levels of prior knowledge. 


3.53 


1.01 


3.17 


1.02 


7b. Accommodates students’ different learning rates. 


3.40 


0.86 


2.57 


0.97 


8. Grouping strategies enable students to work together or alone. The teacher: 




8a. Uses in-class ability grouping. 


1.40 


0.86 


1.33 


0.84 


8b. Has a class that is homogeneous in ability. 


2.90 


1.09 


3.33 


0.92 


8c. Uses cooperative learning arrangements. 


1.43 


0.68 


1.40 


0.89 


8d. Bases individual instruction on mastery of skills and/or concepts. 


2.37 


1.33 


1.80 


1.13 


8e. Uses individualized instruction. 


2.47 


1.48 


1.80 


1.06 


Incentives for Learning 


9. The teacher arouses students’ curiosity by: 




9a. Presenting surprising demonstrations. 


1.80 


1.06 


1.43 


0.77 


9b. Relating topics to students’ lives. 


3.60 


1.30 


2.80 


1.54 


9c. Allowing students to discover information. 


2.93 


1.17 


2.27 


1.17 


9d. Presenting intrinsically interesting material. 


2.93 


1.17 


2.83 


1.21 


10. The teacher uses extrinsic academic incentives such as: 




10a. Praise and feedback. 


4.17 


1.18 


3.53 


1.17 


10b. Accountability 


3.90 


0.89 


3.43 


1.07 


10c. Homework checks. 


3.33 


1.42 


3.47 


1.25 


lOd. Waiting for responses. 


3.80 


0.89 


3.43 


1.14 


lOe. Guiding partial responses. 


4.10 


0.85 


3.60 


0.97 


lOf. Tokens and rewards. 


1.87 


1.36 


1.47 


1.04 


lOg. Communicating high expectations. 


3.80 


0.89 


3.07 


1.02 
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QAIT Items 


Grouping 


Academy 


Control 


Mean 


SD 


Mean 


SD 


tOh. Small groups with individual incentives. 


1.23 


0.43 


1.23 


0.63 


lOi. Students encourage one another to achieve. 


1.77 


0.82 


1.90 


1.03 


lOj. Group contingencies. 


1.33 


0.71 


1.33 


0.66 


11. The teacher uses extrinsic behavioral incentives such as: 




11a. Praise. 


3.20 


1.45 


2.60 


1.22 


1 Ih. Tokens and rewards for improvement. 


1.47 


0.86 


1.47 


0.94 


11c. Group contingencies. 


1.37 


0.72 


1.33 


0.71 


12. The teacher provides instruction that is appropriate for students’ abilities: 
Efforts by the student lead to success. 


3.33 


1.40 


2.80 


1.06 


Use of Time 


13. Allocated time: Necessary time is allocated for instruction. 


3.90 


0.80 


3.80 


0.71 


14. Engaged rates: 




14a. The teacher uses effective management. 


4.00 


0.83 


3.80 


0.61 


14b. Students attend to lessons. 


3.50 


0.73 


4.03 


0.49 



The 40 items were grouped into the four main features of the QAIT: quality of 
instruction, appropriate level of instruction, incentives for learning, and use of time. Table 26 
provides descriptive statistical information for each of the four subscales by group (Academy, 
control). 



Table 26: Descriptive Statistics for SSOS-R QAIT Categories by Group 



QAIT Category 


Group 


Acad 


lemy 


Control 


Mean 


SD 


Mean 


SD 


Quality of Instruction 


3.62 


0.65 


3.26 


0.77 


Appropriate Level of Instruction 


2.50 


0.74 


2.20 


0.54 


Incentives for Learning 


2.77 


0.43 


2.44 


0.46 


Use of Time 


3.80 


0.63 


3.88 


0.42 



One-way ANOVAs were computed to determine whether statistically significant 
differences existed within these subscale variables (quality of instruction, appropriate level of 
instruction, incentives for learning, and use of time) by group (Academy, control). One of the 
four ANOVAs did identify a significant difference by group. With a moderate effect size of .74, 
the statistical significance was accompanied by practical significance. See Table 27 for a 
summary of these results. 
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Table 27: One-Way ANOVA Results for SSOS-R QAIT Categories 



Category 


df 


F 


P 


Effect size 


Eavor 


Quality of Instruction 


1,58 


3.80 


.056 


0.50 


Academy 


Appropriate Level of Instruction 


1,58 


3.22 


.078 


— 


— 


Incentives for Learning 


1,58 


8.29 


.006 


0.74 


Academy 


Use of Time 


1,58 


0.32 


.574 


— 


— 



CERC 



The Classroom Environment and Resources Checklist (CERC) assesses the presence or 
absence of indicators of good classroom environments, as well as the visibility and use of a 
variety of instructional resources. A CERC form was completed at the end of each of the 60 
classroom observations. Table 28 presents the percentages for the presence of 16 environmental 
indicators by group (Academy, control). Eor the Academy group, comfortable 
ventilation/temperature (100%); open, risk-free environment (100%); and adequate lighting 
(97%) were the most frequently seen environmental indicators. Least seen was distinct activity 
centers (3%). Eor the control group, comfortable ventilation/temperature (100%), adequate 
lighting (97%), and posted assignments (93%) were most frequently seen; least seen was distinct 
activity centers (0%). 



Table 28: Numbers and Percentages for Presence of SSOS-R 
CERC Environmental Indicators by Group 



Environmental Indicators 


Grouping 


Academy 


Control 


N 


% 


N 


% 


Culturally mediated instruction 


9 


30% 


5 


17% 


Student-controlled classroom discourse 


7 


23% 


4 


13% 


Use of multiracial materials 


8 


27% 


6 


20% 


Use of nonsexist materials 


8 


27% 


10 


33% 


Posted classroom rules 


24 


80% 


26 


87% 


Posted assignments 


22 


73% 


28 


93% 


Cheerful and inviting classroom 


28 


93% 


19 


63% 


Distinct activity centers 


1 


3% 


0 


— 


Adequate lighting 


29 


97% 


29 


97% 


Comfortable ventilation/temperature 


30 


100% 


30 


100% 


Student work displayed 


26 


87% 


17 


57% 


No distracting internal noises/interruptions 


20 


67% 


24 


80% 


No distracting external noises/interruptions 


20 


67% 


23 


77% 


Open, risk-free environment 


30 


100% 


26 


87% 



Table 29 presents the percentages depicting the visibility and use of 18 environmental 
resources by group (Academy, control). Eor both groups, the resources most often seen in the 
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classrooms were a chalkboard and computer (100% for each). For the Academy group, the least 
often seen resource was games and/or puzzles (3%); for the control group, the least often seen 
was student manipulatives/hands-on materials at 3%. For both groups, the most often used 
resource was the classroom chalkboard (73% and 80%, respectively). The Academy group used 
workbooks/activity books, classroom library, map and/or globe, television, and video resources 
least often (0% each); the control group used games and/or puzzles, instructional aids/props, 
student-used equipment, and audio resources least often (0% each). 



Table 29: Numbers and Percentages for Visibility and 
Use of SSOS-R CERC Instructional Resources by Group 



Instructional Resources 


Grouping 


Academy 


Coni 


;rol 


Visi 


bility 


Use 


Visibility 


Use 


N 


% 


N 


% 


N 


% 


N 


% 


Textbooks 


28 


93% 


10 


33% 


30 


100% 


20 


67% 


Workbooks/activity books 


4 


13% 


0 


— 


8 


27% 


4 


13% 


Worksheets/activity sheets 


13 


43% 


11 


37% 


7 


23% 


7 


23% 


Journals/learning logs 


12 


40% 


7 


23% 


13 


43% 


10 


33% 


Classroom library 


18 


60% 


0 


— 


15 


50% 


2 


7% 


Reference materials 


25 


83% 


1 


3% 


30 


100% 


2 


7% 


Map and/or globe 


14 


47% 


0 


— 


11 


37% 


1 


3% 


Games and/or puzzles 


1 


3% 


1 


3% 


4 


13% 


0 


— 


Instructional aids/props 


11 


37% 


5 


17% 


10 


33% 


0 


— 


Science/lab table(s) 


6 


20% 


1 


3% 


6 


20% 


1 


3% 


Classroom chalkboard 


30 


100% 


22 


73% 


30 


100% 


24 


80% 


Student-used equipment 


6 


20% 


5 


17% 


4 


13% 


0 


— 


Overhead projector 


19 


63% 


7 


23% 


26 


87% 


7 


23% 


Television 


27 


90% 


0 


— 


28 


93% 


8 


27% 


Computer 


30 


100% 


5 


17% 


30 


100% 


7 


23% 


Student manipulatives/ 
hands-on materials 


3 


10% 


3 


10% 


1 


3% 


1 


3% 


Audio resources 


6 


20% 


3 


10% 


1 


3% 


0 


— 


Video resources 


7 


23% 


0 


— 


17 


57% 


3 


10% 



SPC 



The SPC measured five standards of effective pedagogy: joint productive activity, 
language and literacy development, contextualization, challenging activities, and instructional 
conversation. Each standard was rated on a 0 to 4 Likert-type scale (0 = not observed, 1 = 
emerging, 2 = developing, 3 = enacting, and 4 = integrating). An SPC form was completed for 
each of the 60 classroom observations. 
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Academy classrooms demonstrated the highest levels of use for contextualization (mean 
= 1.70, SD = 1.02). The standard found to have the highest level of enactment in the control 
classrooms was challenging activities (mean = 1.07, SD = 0.45). Both groups demonstrated 
lowest levels of enactment/integration for the standard of joint productive activity (means of 0.77 
for Academy classrooms and 0.30 for control classrooms with SDs of 0.90 and 0.60, 
respectively). Table 30 provides descriptive and comparative statistical information for each item 
by classroom group (Academy, control). 



Table 30: SSOS-R SPC Category Descriptive Statistics 
and Differences by Classroom Group 



SPC Category 


Academy 


Control 




Comparative Statistics 


N 


Mean 


SD 


N 


Mean 


SD 


df 


t 


P 


Dif. 


d 


Joint Productive Activity 


30 


0.77 


0.90 


30 


0.30 


0.60 


58 


2.37 


.021 


0.47 


0.62 


Language & Literacy 
Development 


30 


1.53 


0.94 


30 


1.03 


0.56 


58 


2.51 


.015 


0.50 


0.66 


Contextualization 


30 


1.70 


1.02 


30 


LOO 


0.98 


57.91 


2.70 


.009 


0.70 


0.71 


Challenging Activities 


30 


1.53 


0.78 


30 


1.07 


0.45 


58 


2.85 


.006 


0.47 


0.75 


Instructional Conversation 


30 


1.50 


0.78 


30 


0.93 


0.64 


58 


3.08 


.003 


0.57 


0.81 



Table 30 also displays the results of t-tests that were computed to compare Academy 
teachers’ and control group teachers’ mean scores on the SPC items. Significant differences 
favoring Academy classrooms were found in each SPC category. Medium to large effect sizes, 
indicating practical meaningfulness in addition to statistical significance, were also found for 
every comparison. 



Measure of Academic Supportiveness and Climate (MASC) 



The four MASC subscales contain 19 (Student Belonging), 6 (Family Expectations), 7 
(Student Academic Efficacy), and 7 (Eamily/School/Student Involvement) items, which 
respondents rated using a scale ranging from 1 to 5 (not at all true to always true). The ratings 
for each subscale were added together and divided by the number of items in each subscale to 
create a mean score for that subscale ranging from 1 to 5. Table 31 provides descriptive 
statistical summaries for the four subscales for the Academy and control students. Control 
students had higher subscale mean scores across all of the scales than did the Academy students. 
Standard deviations were fairly low to moderate for the control students, while the standard 
deviations for the Academy students were fairly moderate to high across the subscales, indicating 
greater dispersion within the scores for this group. 

A statistically significant difference was found on one of the four subscales (Eamily 
Expectations). The difference favored the control students and had a large effect size (Cohen d 
column on Table 31), indicating practical meaningfulness. The remaining subscale differences 
were not significant. 
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Table 31: MASC Subscale Descriptive Statistics and Differences by Student Type 



Subscale Name 


Academy Students 


Control Students 


Comparative Statistics 


N 


Mean 


SD 


N 


Mean 


SD 


df 


t 


P 


Dif. 


d 


Student Belonging 


12 


3.47 


0.80 


12 


3.62 


0.68 


22 


0.48 


.64 


-0.14 


- 


Family Expectations 


12 


4.28 


0.59 


12 


4.71 


0.30 


16.38 


2.24 


.04 


-0.43 


1.11 


Student Academic Efficacy 


12 


3.75 


0.57 


12 


3.76 


0.65 


22 


0.05 


.96 


-0.01 


- 


Eamily/Sctiool/ 
Student Involvement 


12 


3.59 


1.10 


12 


3.84 


0.78 


22 


0.64 


.53 


-0.25 


- 



Student Attitudes Toward Self and School (SASS) 



The four SASS subscales contain differing numbers of items: Teacher-Centric consists of 
nine items, Self-Efficacy contains 11, Leadership has 10, and Like School is composed of nine. 
Respondents rated SASS items using a scale ranging from 1 to 5 (strongly disagree to strongly 
agree). The ratings for each subscale were summed and divided by the number of items in each 
subscale to create a mean score for that subscale ranging from 1 to 5. Table 32 provides 
descriptive and comparative statistical summaries for the four subscales for Academy and control 
students. Academy students had a slightly higher mean than control students on one scale 
(Teacher-Centric), and the comparison sample of students had slightly higher means on two 
scales. The groups’ means on the fourth scale. Like School, were fairly equivalent. Standard 
deviations were fairly moderate to high across all of the subscales, indicating greater dispersion 
within the scores for both groups. No statistically significant differences were found on any of 
the four subscales. 



Table 32: SASS Subscale Descriptive Statistics and Differences by Student Type 



Subscale Name 


Academy Students 


Control Students 


Comparative Statistics 


N 


Mean 


SD 


N 


Mean 


SD 


df 


t 


P 


Dif. 


d 


Teacher-Centric 


12 


3.66 


0.95 


12 


3.36 


0.93 


22 


0.78 


.44 


0.30 


- 


Self-Efficacy 


12 


3.88 


0.82 


12 


4.06 


0.47 


22 


0.67 


.51 


-0.18 


- 


Leadership 


12 


3.36 


0.81 


12 


3.78 


0.66 


22 


1.39 


.18 


-0.42 


- 


Like School 


12 


2.99 


1.15 


12 


2.93 


0.87 


22 


0.16 


.88 


0.06 


- 



Faculty Interviews 



Professional Development 

The faculty interviews explored faculty members’ perceptions of the nature and 
effectiveness of the professional development activities in which they participated. During the 
summer of 2004, faculty received special professional development about such topics as 
culturally responsive instructional strategies, assertive discipline, classroom management, and 
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various technology-related skills. According to the original CHANGES proposal, professional 
development for the teachers was to take place two weeks before the start of school and focus on 
Reading Across the Curriculum, Writing to Learn Across the Curriculum, Culturally Relevant 
Instructional Strategies (CRIS), Embedded Study Skills, and Graphic Organizers. While the 
teachers did recall focusing on these skills, they also explained that their professional 
development experience covered many other skills and concepts. 

In particular, the faculty described working on “Marzano strategies” (see, for instance, Marzano, 
1998, 2000, 2003; Marzano, Gaddy, & Dean, 2000; Marzano, Pickering, & Pollock, 2001). At 
least two faculty members, including the support teacher, traveled to St. Louis to learn Marzano 
strategies, and the support teacher facilitated the CHANGES professional development on these 
strategies on her return. Other professional development topics teachers remembered addressing 
included developing parent involvement skills, reviewing the expectations and goals of the 
program, learning how to make students comfortable, learning how to garner community 
involvement, honing classroom management skills, and reading research to enhance instruction. 
The faculty members agreed that professional development extended beyond their summer work. 
Specifically, they referred to their common planning time (described further in the next section), 
which occurred everyday, as a type of ongoing professional development. In addition to their 
daily common planning time, they also met one Saturday a month from 9 a.m. to 3 p.m. to 
review and refine their teaching skills. 

Most of the faculty members believed that their professional development prior to the 
opening of school was effective and useful. They explained that the skills acquired directly 
affected their instruction, helped them grow as teachers, and provided necessary motivation for 
initiating a new program. However, the faculty expressed mixed feeling about the common 
planning time and Saturday meetings. They held that it was often useful but that some meetings 
or activities could have been eliminated. Ultimately, most of the teachers interviewed (4 out of 
the 5) had positive experiences with professional development both before and during the school 
year. 



Curriculum and Instruction 

According to the initial proposal, the curriculum for CHANGES was intended to consist 
of the required core courses and appropriate elective offerings. Study skills were to be embedded 
in the core courses, and some of the curriculum was to be interdisciplinary. During the 
interviews, when the teachers discussed and described the CHANGES curriculum, they often 
referred to more than the course offerings. They most often defined the purposes, content, 
activities, and organization of the CHANGES program (Walker & Soltis, 1997). As a result, 
while the courses offered in the Academy were largely the same types of courses taught in the 
regular high school, most of the teachers perceived the curriculum to be much different. 

All of the teachers interviewed believed that the Academy curriculum was more rigorous 
than that of the rest of the school because the CHANGES students were held to higher standards. 
The goal of the program, according to one teacher, was to move students from average to above 
average performance. Most teachers perceived that mathematics was the most rigorous subject. 
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In addition to curricular rigor, some teachers also pointed to instructional strategies as 
being significantly different in CHANGES. One teacher perceived that their “curriculum was 
implemented by four teachers in core classes using best methods.” In Capital High School, she 
held, some teachers used best methods; however, most did not. Additionally, the teachers 
believed that flexibility in scheduling made the curriculum different. It allowed for field trips, 
presentations, and other special activities. 

Another fundamental aspect of the Academy curriculum and instruction mentioned by all 
of the teachers was the common planning time. Most teachers referred to it both as a type of 
professional development and as an aspect of the curriculum. Five days a week, the entire 
CHANGES faculty met for 45 minutes for lesson study, research, discussion, and other planning 
activities. The primary focus was lesson study, a Marzano strategy in which one teacher presents 
a lesson and the other teachers provide evaluation and critique. On some days, the CHANGES 
faculty looked at articles, reviewed test score data, or discussed particular students’ progress and 
difficulties. On Wednesdays, they maintained parent contact by writing good-news cards and 
making phone calls. Most of the teachers said they enjoyed the process. However, some teachers 
perceived that some of the more experienced teachers did not always like it. One of the more 
experienced teachers confirmed that notion, claiming that “the common planning time did 
become redundant after a while ... it could be more effective if they did it less often, especially 
at the end of the year.” 

Curricular rigor and instructional strategies, as noted above, aligned the Academy 
curriculum with the goals of CHANGES, according to all of the faculty members. However, 
some courses were believed to be more aligned with the goals than others. Mathematics classes 
were considered to be less aligned than the other classes, as were reading classes. However, 
reading may have been less aligned because the Academy did not get the reading program that 
program planners initially wanted. 



Alternative Schedule 

The alternative schedule was a marked difference between CHANGES and the rest of the 
school. The CHANGES students had 70-minute and 30-minute classes on an alternating 
schedule. All of the class changes were managed without a bell to alert teachers and students to 
the beginning or end of classes and breaks. According to one teacher, the modified block was 
intended to be used in the following manner: the 70-minute time was to be a “lab,” during which 
teachers could facilitate cooperative learning activities and teach skills or concepts in depth. The 
30-minute class was primarily for revisiting skills or conducting mini-lessons. 

While most teachers believed that the alternative schedule was an effective teaching tool, 
some believed that it was not used effectively in all courses. Teacher B felt that some of the 
“older, more experienced teachers had difficulty with the 30 minutes because they tried to 
complete full lessons.” Similarly, she observed that some teachers treated the 70 minutes as if it 
were a regular period, which resulted in the loss of students’ interest. One reason, aside from 
those pointed out by Teacher B, that teachers may have used their time this way is related to how 
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a teacher perceives her course. Some teachers, for example, felt that their classes were not suited 
for labs and that the 30-minute class did not permit enough time to revisit skills. In contrast, 
other teachers thought that the 70-minute lab created a perfect environment to work on skills and 
concepts as a group, and to provide immediate feedback for students’ skill development. These 
teachers may also have reserved the 30-minute classes to work on mini-lessons. 

Regardless of the teachers’ personal opinions about the alternative schedule, they all 
agreed that the students did not like the schedule. The teachers perceived that the students often 
did not get to see their friends and occasionally missed opportunities to participate in extra- 
curricula activities because of their schedule. Consequently, as Teacher D explained, “Students 
felt they were different, and they didn’t like being separated from the others.” 

One commonly held belief was that the schedule disgruntled students and ultimately 
contributed to students’ tardiness. Certainly, the teachers each acknowledged that the lack of a 
bell exacerbated the problem, but they perceived that students’ dislike of the schedule was the 
main factor. Therefore, to accommodate the students and to create an atmosphere more 
conducive to teaching and learning, the teachers changed the schedule in the spring. CHANGES 
staff maintained the block schedule but matched the students’ break time with that of other 
students in the school. Additionally, students had homeroom at the same time as the rest of the 
school; therefore, they could hear announcements about extracurricular activities and watch the 
closed circuit television program that all other students watched during homeroom. All of the 
teachers believed that the revision to the schedule created a better working and learning climate, 
diminished tardiness, and increased students’ trust of teachers. 



Parental Involvement 

Parental involvement was planned to be a critical aspect of CHANGES. Academy staff 
were to work and communicate directly with parents to inform them of their children’s progress 
and difficulties, and to encourage parents’ to influence their children to work to high standards 
and stay focused on school work. Academy faculty seemed dedicated to achieving this goal. 
Before the school year began, the CHANGES faculty organized a picnic for the students and 
their families. To ensure a good turnout, they provided transportation to and from the picnic. 
When they organized a similar event in October, they had much less participation. One teacher 
concluded that involvement in activities was best when transportation was provided. In addition 
to events, teachers used their common planning time every Wednesday to contact parents by 
phone and mail. Some teachers were moderately successful with reaching parents by phone. 
However, some had difficulty because, as they perceived, parents often moved or changed phone 
numbers. 

All of the teachers believed that the level of parental involvement and support that 
resulted from traditional encouragement activities (e.g., phone calls) was almost the same as the 
level of involvement of parents whose children were not enrolled in the Academy. However, 
they also believed that CHANGES garnered a greater degree of parental involvement through 
“nontraditional” activities. Teacher A explained: 
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Parents came to school whenever their children were having 
problems with grades or teachers. They sought for their children — 
psychological help, anger management programs, and health 
information whenever the counselor suggested it. They followed 
through on all of the “extra” things the students needed. 

Ultimately, the teachers agreed that parents were supportive when they were contacted. The 
parents came to the school when they were asked, even the parents of students who often had 
discipline problems. However, some thought that the level of parental involvement was not 
sufficient. 



Challenges for the CHANGES Faculty 

Virtually all of the teachers interviewed expressed pride in the accomplishments of the 
CHANGES program. However, they also described some of the more challenging aspects of 
teaching in the program. Teachers offered insights into the difficulties in their classrooms, as 
well as their thoughts on problems that resulted from an early shift in leadership roles in the 
program. 

Teachers reported a variety of challenges. One teacher, for example, explained that 
gamering complete teacher buy-in was a significant challenge early in the year because veteran 
teachers did not want to change. Other teachers felt that the students’ perceptions of the program 
posed a challenge. (Some students thought that CHANGES was a special education program and 
resented being associated with it.) Yet most often, and most adamantly, faculty members cited 
classroom management and behavioral problems as the most challenging aspect of CHANGES. 

The behavioral problems ranged from extreme tardiness to tremendous disrespect toward 
teachers. Ultimately, at least 14 students (out of 70) were removed from the program because of 
poor behavior, and of those, 8 were sent to night school by the county board of education. Some 
teachers attributed their problems managing the students, in part, to the following: 

• Teachers had difficulty communicating across cultures with a highly diverse student 
population. This suggests that their professional development on culturally responsive 
instructional techniques was not always effective. 

• Teachers perceived the alternative schedule to cause tardiness and create a feeling of 
separateness among the Academy students. 

• Selection of students for the program was inappropriate. According to the proposal, 
students were to be selected to participate in the Academy based on their seventh- or 
eighth-grade achievement test scores. The academy was to be composed of 60 to 75 
new freshmen students whose achievement test scores fell between the 35* and 49* 
percentile. However, the teachers believed that the program selected several students 
whose scores fell below the 35* percentile, as well as students who had major 
behavioral problems, at least two of whom were on parole. 
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Although teachers believed that each of these factors may have played a role in their challenges, 
most of the teachers agreed that the behavioral problems were a direct result of problematic 
leadership. 

As described previously, early in the year, the Academy administrator left CHS for 
administrative reasons. Therefore, according to Teacher C, the Academy program developed a 
shared leadership model, in which “all teachers had some voice,” and in which the guiding 
teacher and the counselor were usually at the helm. While all of the teachers perceived that their 
leaders did a decent job, most felt that “they had too much responsibility to be very effective.” 

By the second semester, a retired vice principal joined the staff to replace the administrator who 
left the program. He was there, according the teachers, no more than 3 days a week because, as a 
retired educator, he was only permitted to work a certain number of hours. Despite these 
attempts to compensate, some teachers reported that the program never fully recovered from the 
early change in leadership. According to Teacher D, had they had a vice principal from the onset, 
“it would have been a better program because as it stood, teachers didn’t have the authority to 
properly discipline the students.” Additionally, some teachers perceived that the new vice 
principal was not very effective. Teacher D continued, “Kids knew when he was there and used 
that knowledge to their advantage. Usually kids did not respond well to the teachers’ discipline. 
This grew worse throughout the year.” Leadership, according to the Academy faculty, is critical 
for maintaining a manageable school climate, and strong leadership was somewhat lacking in 
CHANGES. 
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SECTION 5: CONCLUSIONS AND RECOMMENDATIONS 



The following section presents discussion of conclusions regarding the implementation of 
the CHANGES initiative during the 2004-2005 school year. Each study theme is addressed in 
turn. Eurther, recommendations for future implementation of this or similar initiatives are 
outlined, as are recommendations for future evaluations. 



Conclusions 



Strategies and Processes Employed in Academy Classrooms 

Academy classes made use of a variety of instructional strategies and classroom 
processes throughout the school year. Teachers’ implementation logs and classroom observations 
both revealed that setting and clearly stating goals and objectives for lessons was the most 
frequently used strategy. Teaching lessons based on state learning standards was the second most 
frequently employed strategy and was thought by teachers to be very effective. Given state and 
local curricula and requirements, we would expect that teachers would make great use of 
standards-based instruction and goal setting for lessons. Thus, this finding was not unexpected. 

CHANGES faculty reported that they made great use of instructional strategies that 
involved interaction with students. Strategies such as questioning or cueing students to check for 
understanding, active listening, giving feedback about student progress, and recognizing student 
effort were all reported in more than 90% of teachers’ implementation logs. Active listening, the 
fifth most frequently used instructional strategy, was rated as the most effective, with very little 
variation among the teachers, as evidenced by the relatively small standard deviation. The other 
instructional strategies were also given fairly high ratings for effectiveness. Thus, CHANGES 
teachers felt that interacting with students through such methods was well received by the 
students and had a positive influence on classroom teaching and learning. 

Recognizing and reinforcing student achievements and efforts in the classroom was both 
a frequent instructional strategy and a frequent classroom activity. Achievement recognition 
activities occurred in 85% of the weeks reported in teachers’ implementation logs. 

Reinforcement and recognition as an instructional strategy was noted in 94% of teachers’ logs. In 
a related observation. Academy teachers frequently used incentives for student learning such as 
praise and feedback. Teachers’ comments and reflections indicated that when students performed 
well academically (e.g., high grades on report cards), they seemed very proud of themselves and 
bragged about their accomplishments. Teacher praise and recognition of achievement may have 
added to students’ sense of pride in their academic success. 

One way in which teachers might have recognized student effort and achievement was by 
posting or displaying student work. In 56% of their implementation logs, teachers noted that they 
had displayed student work during the week. However, student work displays were observed in 
87% of classrooms during observations. The discrepancy in the rate with which teachers 
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acknowledged posting student work and the rate with which it was observed in their classrooms 
is probably inconsequential. Perhaps teachers posted student work for display for long periods of 
time, or perhaps their posting of student work coincided with the classroom observation 
schedule. 

CHANGES teachers reported that they used a variety of strategies and approaches to 
illustrate concepts. Use of graphic organizers and nonlinguistic representations of concepts, 
reported in 79% of implementation logs, was frequently mentioned in teachers’ comments and 
reflections. Comments indicated that students generally responded well to the nonlinguistic 
representations and graphic organizers. Although some students may have had difficulty with 
them at first, the majority of teacher comments indicated that the students liked these strategies 
for learning. The success of these instructional strategies may depend on students becoming 
accustomed to the different ways to represent concepts and organize information. 

Another strategy frequently mentioned in teacher comments and reflections was 
cooperative learning groups. Although not observed frequently during classroom observations, 
teachers reported using cooperative learning arrangements in 66% of weeks reported in 
implementation logs. They were rated as an effective strategy, although the standard deviation of 
0.89 indicates that there was quite a bit of variability in effectiveness across the logs. Teachers 
often commented on the success or difficulties they saw when using student learning groups. 
Occasionally, teachers noted that students did not work well with other group members because 
of personal conflicts, or that they spent a great deal of time socializing rather than focusing on 
the learning task. Teachers also frequently mentioned, however, that students were eager to work 
in groups, worked very well together (even when the members of the groups were not friends), 
and helped each other learn and understand concepts. Thus, cooperative learning arrangements 
were greeted with mixed results in Academy classrooms. Effective implementation of this 
strategy may depend on careful selection of group membership, cooperatively established 
classroom norms for group work, or close teacher or aide supervision to ensure that group time is 
spent on task. 

Discipline issues were frequently mentioned in teacher comments and were noted as 
frequent events in CHANGES classrooms (70% of teacher logs). Teacher comments indicated 
that students were frequently tardy or acted out in response to changes in schedules (e.g., delayed 
schedules, holiday breaks). Disciplinary interventions, however, were the fourth most frequently 
used classroom management strategy (noted in 63% of logs). It is not clear, based on available 
data, what management strategies other than disciplinary interventions were used to address 
discipline issues in CHANGES classrooms. In their implementation logs, some teachers 
mentioned the need for consistent disciplinary interventions both in classrooms and throughout 
the Academy in general. Although classroom observers noted that Academy teachers used 
effective management strategies, it may be important to revisit the issue of discipline and 
establish consistent guidelines both for student behavior in the classroom and in the Academy. 

Instructional strategies that were used least often in the Academy were minimally 
interactive practices such as note taking or summarizing and homework and practice. Homework 
and practice, when used, was rated least effective of all instructional strategies. Teacher 
comments indicated that students often did not complete homework assignments, thus hindering 
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the effectiveness and utility of such a strategy. The two strategies used least often were asking 
students to compare, contrast, classify, or use analogies or metaphors, and generating or testing 
hypotheses. These strategies, despite their infrequent use, were rated highly in terms of 
effectiveness. Teachers in all subject areas employed the strategy of comparing/contrasting with 
fairly equal frequency. Of the 53 instances of generating or testing hypotheses, however, 46 
occurrences were noted in the areas of mathematics and science. When teachers in other areas 
used this strategy, they consistently rated it as effective. It is not clear why teachers in the non- 
math/science subject areas did not make more frequent use of hypothesis generation or testing. It 
could be that teachers in these areas (e.g., English, history) perceive that this strategy is most 
effective for subject areas or topics other than their own. 



Differences Between Academy and Control Classrooms 

Academy and control classrooms were fairly similar for many factors measured during 
classroom observations. However, one important difference between the Academy and the rest of 
Capital High School’s ninth-grade program is the subjects taught. In the Academy, students take 
a reading course as part of their required course work, but ninth-grade students at CHS are not 
required to take a reading class. Thus, eight reading classes were observed in the Academy, and 
only two reading classes were observed in the control group. 

In terms of student engagement, there were fairly similar rates of behavior across both 
groups. Students in the control group, however, were on task significantly more often than their 
peers in the Academy. Most CHANGES students were on task (77%), but the rate was not 
comparable to the control group (87%). It is important to remember that in observations using 
the SSOS-R, students who are not on task may be classified as being waiting (e.g., for the next 
task, for assistance), out of the room, or off task (e.g., specifically engaged in some off-task 
behavior). Off-task behavior and the number of students waiting did not differ between the two 
groups. The Academy, however, experienced a significantly higher rate of students being out of 
the room. This observation could be attributed to the “bell schedule” of the Academy, which did 
not have bells to sound the beginning and end of classes. Rather, students relied on classroom 
clocks or their watches to get to class on time. The lack of bells sounding could have contributed 
to students’ tardiness (frequently mentioned in teacher and counselor implementation logs), 
which would have been revealed as students being out of the room during SSOS-R observations. 

SSOS-R observations permit evaluators to examine the groupings and activities of 
students and teachers in the classroom. Teachers, aides, and students can be involved in 
interactions, working alone, performing management activities, or engaging in social activities 
(or in other ways be uninvolved in the activities of the class). Academy and control classrooms 
were comparable in the number of students involved in interactions with other students, students 
working alone, and students being social or uninvolved (i.e., there were no differences between 
the groups). In the control group, however, teachers were involved in interactions and 
management activities with significantly more students than teachers in the Academy. Thus, 
more students were working interactively with teachers in control classrooms. This could 
indicate that teachers in Academy classrooms were more often working with individuals or small 
groups of students rather than providing instruction or lecture to the class as a whole. 
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The COF instrument of the SSOS-R permits observers to record the number of minutes 
spent per observation period in each of 28 discrete activities. The 28 activities can then be 
grouped into four categories (teacher led, student led, management/organization, and off task; 
see Table 22), and a mean number of minutes spent on each activity category can be computed. 
In this evaluation, observers focused alternately on the teacher and on a target student. When the 
focus of the observation was the target student (i.e., the observer was recording the number of 
minutes the student spent engaged in activities), only one significant difference emerged. 
Students in control classrooms spent more time on teacher-led activities during the student focus 
of observations. When the focus of the observation was the teacher. Academy classrooms spent 
significantly more time on teacher-led activities and on management or organization. 

The difference between the classroom groups in minutes of teacher off-task behavior was 
noticeable, but not significant, in that Academy teachers spent 1.75 minutes per hour off task and 
control group teachers spent slightly more than 4 minutes an hour off task. Combining the mean 
number of teacher and student minutes spent off task reveals that Academy classrooms spent 
2.36 fewer minutes per hour engaged in off-task activities than did their control classroom peers 
(7.56 minutes versus 9.92 minutes, respectively). The difference can be attributed to Academy 
teachers’ low number of off-task minutes. This finding indicates that Academy teachers were 
very successful at staying focused on appropriate tasks during class time, even more so than their 
control group peers. 

Despite the slight differences in the number of minutes teachers spent off task, observers 
rated both Academy and control group classrooms equally well in terms of use of classroom 
time. Teachers in CHANGES classrooms, however, were rated significantly higher than those in 
control classrooms in terms of the use of incentives for learning. CHANGES teachers were rated 
almost significantly better than their peers in terms of quality of instruction (the p value for the 
test was .056, just higher than the established alpha level of .05; see Table 27). Academy 
classrooms were also rated slightly better in terms of appropriate level of instruction. These 
findings indicate that Academy teachers were more successful than their control group peers in 
delivering high-quality instruction and providing appropriate motivation and encouragement for 
their students to learn. They were also slightly more successful at gauging student learning needs 
and providing instruction that suited those levels. 

CHANGES faculty were also more successful than their control group peers in creating 
classroom environments that demonstrated equity as well as sensitivity and responsiveness to 
students’ cultures. In comparison to their control group peers. Academy teachers used more 
culturally mediated instruction, more student-controlled discourse, and more materials that were 
multiracial. Additionally, Academy classrooms were much more likely to be cheerful and 
inviting and to have student work displayed around the room; all of them demonstrated open and 
risk-free environments. These findings all indicate that, to a much greater degree than control 
group teachers, CHANGES faculty were successful in creating equitable, inviting, and safe 
learning environments for students. 

Another indicator of Academy teachers’ success in creating culturally responsive 
teaching and learning environments is evidenced by observation data collected using the SPC. 
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The five criteria that compose the SPC are markers of effective pedagogy and, according to 
Tharp and colleagues (2000), are extremely effective strategies for teaching students who are 
culturally, linguistically, and economically diverse and who may not be as successful in school 
as some of their peers. Academy teachers enacted the criteria of the SPC significantly more than 
their control group peers. That is, to a much greater degree than teachers in other CHS 
classrooms, CHANGES teachers worked closely with their students (e.g., via joint productive 
activities) to achieve challenging learning goals. Academy teachers were more successful in 
making their content relevant to students’ lives and experiences (contextualization) and in 
engaging students in instructional conversations that, in part, could help students’ develop better 
language skills. This greater implementation of effective pedagogy in CHANGES classrooms is 
promising and may have helped students learn more than they might have, had they not 
participated in CHANGES. Eurther, the close relationships between teacher and student that 
these teaching strategies may have engendered may have helped students feel that they were 
important to their teachers and that their teachers had high expectations of them (see the 
discussion of student attitudes below). 

Given that Academy teachers noted a high degree of tardiness among their students, and 
given that class movement bells for the regular school population rang during Academy class 
time, one might expect that there would be a higher level of internal and external distractions and 
interruptions in Academy classrooms. This expectation was confirmed by classroom observation 
data. Control classrooms were more likely than CHANGES classrooms to have no distracting 
noises or interruptions, either external or internal. Thus, Academy classrooms had a greater 
prevalence of distractions and interruptions. 

Differences in the rate with which Academy and control teacher used various resources 
may further indicate instructional differences. Academy teachers were observed to make much 
less use of textbooks, workbooks, television, and video resources and much greater use of 
activity sheets, games, props or other instructional aids, and student manipulatives or hands-on 
materials. These differences could indicate that Academy teachers relied less on the traditional 
methods of presenting material (e.g., textbook readings, movies) and made greater use of games 
and other hands-on methods for engaging students in the presentation of instruction. 

CHANGES was designed to encourage teachers to use a variety of instructional strategies 
and methods of presenting material to the students. The purpose of encouraging teachers to use a 
variety of strategies, presumably, was to increase the likelihood that most or all students would 
respond to some method of presentation; become engaged in the learning; and, consequently, 
learn more in the CHANGES classroom. The findings of this evaluation indicate that Academy 
teachers did use a variety of strategies and that, in some ways, those strategies were different 
from those used in the regular CHS ninth-grade classrooms. However, because Edvantia 
evaluation staff did not have access to student academic outcome data, no conclusions can be 
made about the effectiveness of those strategies in comparison to those used in other CHS 
classrooms. 
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Differences Between Academy and Control Students’ Attitudes and Perceptions 

Edvantia staff assessed the attitudes and perceptions of both Academy and control 
students. In general, there were no differences between CHANGES and control students in terms 
of their perceptions about their school and families’ supportiveness and climate for learning. One 
notable exception relates to familial expectations. CHANGES students reported that they felt like 
their families had lower expectations of them in their academic endeavors than control students 
reported. The difference was very strong. These students also reported slightly lower perceptions 
that their families and schools were working together with them in their education. In teacher and 
counselor implementation logs, comments often hinted that students might have little support at 
home. It may be worth noting, however, that CHANGES students generally rated “Eamily 
Expectations” higher than the other three scales addressing students’ perceptions (MASC). 
Therefore, it may be reasonable to conclude that students in the Academy did think that their 
families had high expectations; they just did not perceive those expectations to be as high as 
expectations perceived by their control group peers. 

It may also be important to note that there were no differences between control and 
CHANGES students in terms of perceptions of academic self-efficacy and belongingness to their 
schools. Therefore, we can conclude that Academy students felt as confident as their control 
group peers about their ability to do well academically. This finding is meaningful because 
teachers stated that some students resented being in the CHANGES program because they 
thought it was a special education program. If students believed that they were being placed into 
a program for “special students,” or that their school or administration did not have confidence in 
their ability to do well in school, it could have become a self-fulfilling prophecy in which 
students “lived down” to the low expectations they thought their school had for them. The 
finding of no difference on this scale, then, indicates that the students did not believe that they 
could not succeed in school. The lack of a difference could indicate a kind of defiance in 
Academy students (e.g., “You placed me in this program because you think I can’t do well, but I 
know that I can do well.”). It could also indicate that CHANGES teachers were effective in 
expressing high expectations for their students and communicating that they believed students in 
the Academy were capable of doing well. 

Students in the Academy and control group also did not differ in their attitudes toward 
themselves and school. Control students had slightly more confidence in their ability to identify 
problems and take action; further, they had a slightly better perception of themselves as leaders. 
Academy students had a slightly more positive perception of their teachers’ supportiveness and 
attitudes toward students. That is, in comparison to the control group students. Academy students 
felt more like their teachers had positive attitudes toward them and supported their academic 
endeavors. This finding suggests that Academy teachers were somewhat successful in 
demonstrating that they cared about how well their students performed in school. These 
differences were not significant, however. Eurther, Academy and control group students did not 
differ in terms of how much they enjoyed learning or took ownership of their learning; neither 
group seemed to have a high degree of liking for school. Therefore, we can conclude that 
Academy and control students were not very different from one another in terms of their 
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perceptions and attitudes about themselves and school. The only exception is their perceptions of 
their families’ expectations of them, discussed above. 



Summary 

The purpose of this study was to examine the implementation of CHANGES. Evaluators 
discovered that, consistent with the intentions of the initiative, CHANGES faculty made use of a 
variety of instructional practices. All teachers and the counselor had regular contact with parents 
of CHANGES students. Teachers seemed to express a greater degree of comfort interacting with 
culturally diverse students in their classrooms than with culturally diverse parents. CHANGES 
teachers’ classroom practices were of higher quality and were significantly more culturally 
responsive than those of their counterparts in regular classrooms. However, students in regular 
classrooms were more often on task. Discipline issues were frequent concerns of Academy 
faculty; inconsistent procedures and lack of a central administrator devoted solely to CHANGES 
may have exacerbated discipline disruptions. At the end of their freshman year, CHANGES 
students were remarkably similar to their control group peers in terms of perceptions and 
attitudes about themselves and their educational experiences. Eurther, CHANGES students felt 
slightly more like their teachers had positive attitudes toward them and supported their academic 
endeavors. Although this study had methodological challenges, findings suggest that the 
CHANGES initiative was successful in promoting a positive ninth-grade classroom experience 
for the students involved in the program. Eurther study is needed, however, to assess the impact 
and effectiveness of the Academy on student academic outcomes. 



Recommendations 

Based on the findings of the implementation study, and based on the conclusions drawn 
from those findings, several recommendations are offered both for future implementation of 
CHANGES or similar initiatives and for future evaluations. Recommendations are presented first 
for future implementations of the CHANGES program. Recommendations for future evaluations 
of similar programs follow. 



Recommendations for Future Implementation 

Eirst, and perhaps most importantly, the Academy needs to have a full-time 
administrator. The removal of the first administrator early in the year impacted the entire 
implementation. Although CHANGES faculty tried to cope with the change and a part-time 
administrator was employed during the second semester, the absence of a full-time administrator 
who was completely devoted to CHANGES most likely affected both students and teachers in 
ways we may not be able to measure. A full-time administrator would be able to monitor and 
deal with Academy-wide discipline issues, support teachers’ classroom management policies, 
and follow up on contacts with parents about both positive and negative occurrences in school. 
Eurther, the administrator would be able to handle both routine and special requests (e.g., from 
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the evaluators), thereby allowing the Academy counselor and support teacher to devote their time 
to their stated responsibilities. 

Although a full-time administrator will resolve some of the discipline concerns expressed 
by CHANGES faculty during the 2004-2005 school year, the Academy should establish a 
discipline or behavior management plan from the outset. Responses to teacher and counselor 
implementation logs indicated that discipline issues were handled as they arose and that there 
might not have been consistency throughout the Academy or over time. A specific plan for 
dealing with discipline issues consistently should be in place at the outset of future 
implementations. Such a plan may also eliminate the strategy of using field trips as behavioral 
rewards; that strategy was a concern for many teachers because often, those students who were 
not eligible for them may have benefited the most from them. Rather, field trips might be 
planned to include all students, regardless of past behaviors. 

CHANGES faculty and staff need continued support for culturally responsive instruction 
and interactions with students and their parents, bindings indicate that teachers were successful 
in using culturally responsive instructional strategies and creating responsive learning 
environments for their students. Teachers’ interview comments, however, suggest that they still 
felt like there were difficulties in communicating with the parents of such a diverse student 
population. Therefore, resource teachers or consultants should continue to provide support for 
culturally responsive instruction; further, their responsibilities should be broadened to include the 
provision of support for culturally responsive interactions with the parents of diverse students. 

To the extent possible, CHANGES and CHS faculty should encourage family 
involvement in students’ schooling. Attempts were made to involve parents during the 2004- 
2005 school year, and those efforts were met with mixed success. However, because CHANGES 
students felt a slightly lower sense that their families and schools were involved in their 
schooling and a significantly lower sense that their families had high expectations of them (when 
compared to similar control students), it is important to try to foster as much familial 
involvement and investment as possible. Perhaps one way to achieve this goal is to provide 
regular and frequent updates to parents about their child’s performance. Teacher implementation 
logs indicated that updates to parents seemed to motivate some students to apply themselves 
more to their academic work. Euture implementations could build on this strategy and provide 
regular updates to parents about students academic work, behavioral performance, school 
activities, and so on. Included with these updates could be explicit invitations to attend field 
trips, presentations, classes, or other special school activities. 

Teachers should also continue to use interactive instructional strategies. Teacher 
comments indicated that students, for the most part, enjoyed the interactive activities and were 
very engaged in those activities. Excitement about the activities was often mentioned. Teachers, 
then, can use interactive strategies to capitalize on Academy students’ senses of excitement and 
curiosity. 

In the future, CHANGES staff should be mindful that changes or alterations in students’ 
regular schedules or Academy policies can have a great impact on students. Students were often 
noted to be “hyperactive,” defiant, or generally wound up when there were changes in their 
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regular schedules (e.g., changes to the bell schedule, delays for clubs or assemblies). Although 
schools have no control over some schedule changes (e.g., weather delays or closings), every 
attempt should be made to avoid unnecessary alterations, especially dramatic changes midterm 
or midyear. Expectations should be established at the beginning of the year and adhered to with 
minimal modification. 

Finally, CHANGES and CHS faculty should encourage students to participate in school 
and Academy activities and provide ample opportunity for students to interact with their fellow 
freshmen. Several teacher comments indicated concern that CHANGES students felt separate 
from the rest of the school, and although the differences were not significant, CHANGES 
students did feel slightly less like they belonged to the school than did their control group peers. 
Effort should be made to minimize students’ feelings of isolation from the school community by 
encouraging participation in school clubs, events, and activities. Midway through the 2004-2005 
school year, CHANGES staff attempted to provide opportunities for Academy students to 
interact with students in the general CHS population by changing the bell schedule to align 
students’ break times. Such strategies should be employed from the start of the year in future 
implementations . 



Recommendations for Future Evaluations 

Future implementations of CHANGES or similar initiatives should be evaluated by an 
independent third party. The following recommendations are supplied to help future evaluators 
conduct the most effective evaluation possible. 

First, evaluators and Academy staff need to establish written expectations for both the 
evaluators and CHANGES staff. A memorandum of understanding would suffice to provide all 
parties with written agreements about who is responsible for what activities. The agreement 
should include expectations, timelines, and procedures for timely communication. Further, the 
agreement should provide appropriate guidelines for recourse of action should the parties not 
fulfill their agreed-upon obligations. 

Establishing timely communication will be key for future implementation and will be 
greatly aided by the addition of a full-time Academy administrator. In the current year’s 
evaluation, the removal of the administrator contributed to difficulties in communication. 
Evaluators, for instance, were not notified when the Academy bell schedule changed midyear, 
and often, evaluators’ requests, phone calls, and e-mails were not returned in a timely manner. 
Much of the communication difficulties can be attributed to the lack of a central staff member 
solely devoted to CHANGES administration. 

Evaluators and CHANGES staff also need to establish a more effective strategy for 
obtaining parental consent for participation in evaluation activities. The evaluation was greatly 
impacted by difficulties in obtaining consent from parents both of CHANGES and control 
students. Because of these difficulties, the evaluation had to be redesigned midyear, thereby 
eliminating the proposed pretest and weakening evaluators’ ability to draw strong conclusions 
about the impact of the CHANGES initiative on students’ attitudes, perceptions, and skills. 
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Parental consent must be obtained for all minor students who will participate in research and 
evaluation activities. This important requirement will not change, and all stakeholders in 
evaluation must work together to establish an effective, efficient mechanism for obtaining the 
return of signed informed-consent forms. 

Data collection instruments used in this evaluation were sufficient for answering research 
questions and could be used in future evaluations. Minor changes to the implementation logs for 
teachers and counselors should be made, however, to ensure that instructions are explicit and 
clear. The inclusion of specific instructions will ensure that evaluators and faculty completing the 
instruments have a shared understanding of what is expected and what the instruments are 
intended to accomplish in terms of the evaluation. Classroom observations completed with the 
SSOS-R instruments were extremely useful in examining classroom practices and resources used 
in the Academy. Further, they helped explicate similarities and differences between Academy 
classroom instruction and that occurring in regular CHS classrooms, particularly with respect to 
culturally responsive classroom practices (as measured by the SPC). Evaluators encourage the 
use of the SSOS-R in future evaluations of this type. 

Future evaluations should explicitly examine student academic outcomes. Although CHS 
faculty planned to examine student outcomes, such as GPA and student performance on the 
Terra Nova exam, the evaluation would be strengthened by explicit examination of such 
outcomes. At a minimum, student GPAs should be examined for each reporting period. 
Continued administration of a standardized test such as the Terra Nova would be beneficial. 
Preferably, in addition to GPA and academic testing, other student outcomes, such as 
disciplinary referrals, tardies, and absences, will also be tracked as part of the evaluation. Future 
evaluations should also employ a pretest/posttest design, as originally proposed for this 
evaluation. Of course, such a design is predicated on the presumption that parental consent can 
be obtained for all students in a timely manner. 

CHS and CHANGES faculty may wish to examine the academic outcomes for 
CHANGES participants throughout their time in high school, as consistent with the stated goals 
for the program (e.g., increasing the number of low-performing students enrolling in advanced 
placement and honors courses during their junior and senior years). If this is the case, CHS and 
CHANGES administrators should collaborate with evaluators to establish a long-term evaluation 
plan for data collection, analysis, and reporting. 
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APPENDIXES 




Appendix A: 

CHANGES Teacher Implementation Log 




AEL CHANGES Teacher Implementation Log 



Week: 



Teacher ID: 



This week’s learning goals were: 


How will I know that students learned this 
week? 








Number of parent contacts made during the past week: 






Please check the box to the left of the instructional strategies used during the past week. Please 
indicate the effectiveness of each instructional strategy checked (at left) by bolding or highlighting 
one number from 1 to 5 (at right). 


Instructional strategies used during the past week: 


Not at all Extremely 

Effective Effective 




Set goals or objectives for lessons 


1 


2 


3 


4 


5 




Lessons and/or instruction were related to state learning standards 


1 


2 


3 


4 


5 




Gave information and/or illustrated concepts with a variety of 
approaches/strategies 


1 


2 


3 


4 


5 




Questioned or cued students to check for understanding 


1 


2 


3 


4 


5 




Asked students to compare, contrast, classify, or use 
analogies/metaphors 


1 


2 


3 


4 


5 




Connected new information to prior knowledge 


1 


2 


3 


4 


5 




Active listening to student responses 


1 


2 


3 


4 


5 




Summarizing or note taking 


1 


2 


3 


4 


5 




Graphic organizers or non-linguistic representation 


1 


2 


3 


4 


5 




Learning groups/cooperative learning 


1 


2 


3 


4 


5 




Scaffolds instruction 


1 


2 


3 


4 


5 




Generated or tested hypotheses 


1 


2 


3 


4 


5 




Provided feedback to students about progress toward objectives 


1 


2 


3 


4 


5 




Reinforced and/or recognized student effort 


1 


2 


3 


4 


5 




Homework and practice 


1 


2 


3 


4 


5 




Comments/Reflections about instructional strategies used during the past week: 
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AEL CHANGES Teacher Implementation Log 



Classroom procedures and classroom management 
strategies used during the past week: 


Activities/events that occurred in the 
classroom during the past week: 


Rules, schedule, procedures, etc. posted in classroom 

Clearly stated goals, objectives, and expectations 

Visible/posted Rubrics and/or scoring guides 

Displayed student work/homework in classroom 

Built positive teacher- student relationships 

Disciplinary interventions 

Other {please describe): 


Discipline issues: 

Achievement recognitions 

Speaker or outside presentation 

Student presentation(s) 

Technology time/activities 

Lab(s) 

Other {please describe): 



What events (if any) happened in the school that might affect students (in terms of 
concentration, achievement, emotions, etc.)? 



Reflection: Ideas, insights, suggestions, changes needed: 



Other comments (if any): 



Thank you! 
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Appendix B: 

CHANGES Counselor Implementation Log 




AEL CHANGES Counselor Log 



Week: 



Counselor ID: 



Number of students seen this week: 



Number of parents contacted this week: 



Reasons students sought counseling this week: 



Referrals made this week: 



Academic counseling 

Career counseling 

Home/family issues 

Social problems/issues outside of school 

Conflicts/issues with school staff 

Conflicts/issues with other students 

Other psychological needs 

Teacher referrals 

Other {please describe): 



Student assistance team 

School nurse 

Mental health/psychological services 

Law enforcement 

Social services/DCFS 

Social workers/case managers 

Upward Bound 

Other {please describe): 





What events (if any) happened in the school that might affect students (in terms of 
concentration, achievement, emotions, etc.)? 






Consultation: 






Reflection: Ideas, insights, suggestions, changes needed: 






Other comments (if any): 




Thank you! 
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Appendix C: 

Parental Consent Information and Form 




August 26, 2004 



Dear Parent, 



Capital High School continually looks for new and better ways to improve teaching 
and schooling practices to make sure that all students will have the opportunity to meet their full 
potential. This year, teachers at Capital will be implementing a variety of instructional practices 
aimed at helping ninth graders succeed during their first year in high school. It is important to 
research these practices to find out what works best for the students. 

AEL, a research organization in Charleston, will research the teaching strategies and schooling 
practices used with freshmen at Capital High School during this school year. The purpose of the 
research is to find out what effects various teaching practices have on students’ attitudes and 
achievement during their freshman year. 

Your child has been selected to participate in this important research study. AEL researchers will 
ask your child to complete questionnaires about their attitudes and experiences in the school and 
will observe instruction in your child’s classroom a few times during the year. AEL researchers 
may also look at your child’s academic record for this school year. There are no known risks 
associated with this project that are greater than those ordinarily encountered in daily life, and no 
participants (students, faculty, or administrators) will receive direct compensation for this study. 
Information gathered in this project, however, will be analyzed and shared with Capital High 
School faculty to inform instructional and administrative practices that can benefit your child. 

All information and data gathered during this project will be held in the strictest of confidence. 
Data will be stored in secure locations at the AEL offices in Charleston. Only authorized AEL 
staff members (such as the lead researcher and the research assistant) will have access to the 
information. Data will be kept at the AEL office for a length of time in accordance with industry 
standards (usually about five years). AEL researchers value participants’ confidentiality and will 
take all reasonable precautions to protect your child’s identity and confidentiality. 

Results of this research will be reported to administrators at Capital High School and the 
Kanawha County school board. No information that could identify your child will ever be 
released to the public, and your child will never be identified by name or role in any reports that 
are written about the research. 

Please sign and return the enclosed permission form to allow your child to participate in this 
important research. Your child’s input will be essential to help us discover what strategies are 
most helpful for high school freshmen. 




August 26, 2004 
(Page 2) 



If you have any questions about this research, please contact Georgia Hughes at AEL (347-0413 
or hughesg@ael.org). You may also contact Merrill Meehan, Chair of AEL’s Institutional 
Review Board (347-0432 or meehanm@ael.org), which is charged with protecting the rights of 
our research participants. Please keep this letter for your records when you return the enclosed 
permission form. 

Thank you very much for your time and for allowing your child to participate in this important 
research. 

Sincerely, 



Georgia Hughes 

Research & Evaluation Specialist 

Phone: 347-0413 
E-mail: hughesg@ael.org 




Capital High School - AEL 
Informed Consent Form 
CHANGES Evaluation 



I understand that my child has been selected for participation in a study to examine how 
various instructional strategies affect new freshmen at Capital High School. I understand that 
AEL researchers will visit my child’s classroom periodically during the school year and observe 
instruction. I also understand that individual children will be surveyed occasionally by AEL 
researchers, and that the researchers will review student information such as achievement test 
scores, grade point average, and other relevant records (such as classroom grades). Eindings of 
the study will be shared with policymakers, researchers, educators, parents, and other interested 
persons through various means, such as periodic AEL newsletters, conference papers, journal 
articles, books, and presentations. Names and identifying characteristics will not be used in any 
publication of study findings, and AEL will maintain my child’s confidentiality. I will be given 
the opportunity to review any information that is unique to my child (for instance, direct 
quotation) prior to its publication. 

Information provided for the study will be held in strictest confidence, with the exception 
that if the researcher obtains clear evidence of unlawful behavior that could result in physical or 
mental damage to a minor, the researcher is required by statute to report such evidence to the 
appropriate authorities. 

I understand that I am free to withdraw permission for my child’s participation in the 
study at any time by notifying the teacher, and that there will be no negative consequences from 
AEL as a result of this withdrawal. My signature on this consent form indicates my willingness 
for my child to participate in the study. The researcher’s signature on this form indicates that the 
researcher has given me information about the study. 



Child’s Name: 
Parent or Guardian: 
Signature: 
Date: 

AEL Researcher: 
Date: 




Appendix D: 

Interview Protocol for Faculty Interviews 




CHANGES Faculty Interview Protocol 



The purpose of the interview is two-fold: to assess the extent to which each of the major 
components of the CHANGES program was implemented; and to gauge the degree to which the 
implementation of the project was consistent with the original objectives. 

INTRODUCTORY COMMENTS & QUESTIONS 

How did you become involved with CHANGES? 

How many years of teaching experience do you have? 

How many years have you been at Capitol? 

PROFESSIONAL DEVELOPMENT 

How often did you participate in professional development? 

In what kinds of PD activities did you participate? 

CURRICULUM & INSTRUCTION 

How does the curriculum of the academy differ from that of Capitol High School? 

Consider the goals of CHANGES. In what areas is the curriculum aligned with the goals and 
where is alignment was lacking? 

What, in your opinion, are the biggest challenges of teaching in the academy? 

How did you use the common planning time? 

SCHEDULE 

What did you do differently with the 70-minute period than you did with the 30-minute period? 
What was the rationale for changing the bell schedule? 

PARENTAL INVOLVEMENT 

What kind of support, if any, did you receive from parents? 

LEADERSHIP 



Tell me about the leadership for CHANGES. 





Capital High Academy for Ninth Graders Exceeding 
Standards (CHANGES) 

Faculty Interviews 



Contacts: 

Georgia Hughes, AEL Research & Evaluation Specialist I (304-347-0413; hughesg@ael.org) 



Appalachia Educational Laboratory, Inc. (AEL) in Charleston is collaborating with Capital High 
School to evaluate the Capital High Academy for Ninth Graders Exceeding Standards 
(CHANGES). As part of the evaluation, we are conducting interviews with CHANGES faculty 
such as yourself to find out more about how the program functioned and what impact it had on 
teachers and students. 

We are asking that you give us your honest feedback and opinions about various aspects of 
CHANGES. The interview should require about 30 minutes of your time. Participating in this 
interview should involve no risks to you that are any greater than those you experience in your 
daily life. Your participation in this survey is voluntary; you may discontinue your involvement 
at any time without any reprisal or penalty. 

AEL staff will take every reasonable precaution to protect your confidentiality throughout this 
project . The information that you provide will be combined with information from other 
CHANGES faculty and reported all together. Your individual responses will not be singled out 
and you will never be identified by name or role in any report(s). Only your interviewer (Aaron 
Baker) and Georgia Hughes will have access to your responses. 

Taking part in this interview will allow you to provide important and useful feedback to CHS 
and the CHANGES program, which will use the results of this research to inform future 
implementation of CHANGES or similar initiatives. Although you will not be compensated 
directly for your participation, your thoughts and suggestions will help CHS and CHANGES 
administrators make important decisions about how they can best serve the needs of CHS 
students and faculty. 

Your participation in the interview indicates your consent. If you have any questions or concerns 
about this interview, including its purpose or expectations of you as a participant, please contact 
Georgia Hughes (304-347-0413 or hughesg@ael.org). If you have any questions or concerns 
about your rights as a participant in this research, you may contact Dr. Merrill Meehan, AEL 
Institutional Review Board (IRB) Chair (304-347-0432 or meehanm@ael.org). 



^ Data collected for research purposes are stored in compliance with ISO 17799 requirements for access, security, 
and redundancy. Data are stored in an encrypted format in a centralized, electronically and physically secure server 
at AEL for a period not to exceed five years. All electronic data of a personal nature are safeguarded and available 
only to those project leaders, staff, and technologists having a need to know within the specific criteria as set forth in 
the approved project plan. The AEL Institutional Review Board (IRB) has the authority to inspect consent records 
and data files only to assure compliance with approved procedures. 



Appendix E: 

Administration Instructions for Student Instruments 




Instructions for Administering the AEL AAASC and AEL SASS 



1. The two questionnaires are placed inside the larse envelopes and collated in bundles 
of 1 0. Contact the CHANGES counselins office for additional copies of each if 
necessary. 

2. Administer the questionnaires to students during the arranged time period. Completion 
of the questionnaires should take approximately 20-30 minutes and should be siven 
without interruption. 

3. Students should be instructed not to discuss their responses with others durins the 
administration or thereafter. 

4. Please ask students to fill in the bubbles completely and not to use checkmarks or Xs to 
respond. Also, please instruct them to fill in only one bubble per question . 

5. Students can use either a pencil or a pen to complete the survey. Please ask them to 
use a pen with blue or black ink only, no colors such as red, sreen, etc. 

6. If a student wants to chanse an answer and is usins a pencil to fill out the survey, 
please instruct him or her to erase completely the incorrect response. If a student is 
usins a pen, instruct him or her to put an X throush the incorrect response. If a student 
makes more than one mistake and would like a new copy of the survey, feel free to 
Sive him or her one. Collect and destroy all incomplete surveys. 

7. Tell students not to make any additional marks on the sun/ey. This is a scannable 
document and additional marks can cause problems in the scannins process. 

8. If a student has a question about the meanins of a word contained in a question, you 
may define the word. However, we ask you not to explain or rephrase the entire 
question. In addition, please do not instruct students about how to respond to 
particular questions. 

9. Instruct the students that they are to place their completed sun/eys back inside the 
larse envelopes and seal them. When all students have completed the questionnaires, 
please collect the sealed envelopes and them and place them in one of the larse 
white envelopes provided. Write whether the sroup of students was “Academy” or 
“non-Academy,” and return the envelopes to the CHANGES counselins office. Return 
any blank survey forms in the envelope with the completed surveys. 

A report of compiled survey results will be sent to the 
principal when the analysis is completed. You may want to 
reassure students that no students or teachers will be 
identified by name in the report and that no one will see 
their answers except for the researchers. 




Appendix F : 

Student Assent Information 





AEL 

P.O. Box 1348 

Charleston, WV 25325-1 348 
Phone: 304-347-0400 
Fax: 304-347-0467 



AEL & Capital Hish School 



2005 CHANGES Student Survey 



Capital Hish School is lookins for ways to help freshmen students. We need to find out 
how freshmen like you feel about school and what you think about thinss at your school. 
A company here in Charleston called AEL is doins the study, and a woman named 
Georsia Hushes is the head researcher. Your parents have been told about this study, 
and they have said that it is alrisht for you to participate if you want to do so. 

Today we are askins you to answer some questions about yourself and what you think 
and feel about your school. There are no risht or wrons answers. You do not have to 
answer the questions, but we hope that you will. Your srades will not be affected at all 
whether you decide to answer the questions or not, and you will not set rewarded or 
disciplined either way. Your answers will be combined with other kids’ answers. Your 
answers will be used just for this research project and will never be siven to your 
teachers, your parents, or anyone other than the researchers. The researchers will keep 
your questionnaires in a secure place where no one else can see them. A report will be 
written about the findinss of this study, but your name will never be reported. If you do 
not want to answer a question, you may leave it blank. 

There are two questionnaires for this sun/ey: a sreen one that is two (2) pases Ions and a 
yellow one that is three (3) pases Ions. When you have finished answerins the questions 
on both questionnaires, please place them both back in the larse envelope and seal it. 
Your teachers have been instructed to leave the envelopes sealed so they cannot look at 
your answers. 

If you do not understand some of the directions or questions, please ask your teacher 
for help. If you have any questions about this project, you can call Georsia at AEL. Her 
phone number is 347-041 3, and her e-mail is hushess@ael.ors. 



Thank you for takins the time to fill out these questionnaires! 
We really appreciate your help with this important research! 



Appendix G: 

Evaluation Standards Checklist 




Checklist for Applying the Program Evaluation Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear 
in Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, 
CA, Sage. 



The Standards were consulted and used as indicated in the table below (check as appropriate): 



The Standard 
was addressed 



The Standard was 
partially addressed 



Standard & Descriptor 



Stakeholder Identification 

Evaluation Credibility 
Information Scope and Selection 
Values Identification 
Report Clarity 

Report Timeliness and Dissemination 

Evaluation Impact 

Practical Procedures 

Political Viability 

Cost Effectiveness 

Service Orientation 

Formal Agreements 

Rights of Human Subjects 

Human Interactions 

Complete and Fair Assessment 

Disclosure of Findings 

Conflict of Interest 

Fiscal Responsibility 

Program Documentation 

Context Analysis 

Described Purposes and Procedures 
Defensible Information Sources 
Valid Information 
Reliable Information 
Systematic Information 
Analysis of Quantitative Information 
Analysis of Qualitative Information 
Justified Conclusions 
Impartial Reporting 

Metaevaluation 



The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 



The Standard was The Standard was 
not addressed not applicable 



^ May be metaevaluated 
at a later time. 



□ Request for evaluation plan/design/proposal 

□ Evaluation plan/design/proposal 
□_ Evaluation contract 

Name: Georgia K. Hughes 

(typed) 



0 Evaluation report 
□ Other: 



Date: September 6, 2005 



(signature) 

Position or Title: Research & Evaluation Specialist 1 

Agency: Edvantia, Inc., Institute for the Advancement of Research in Education 

Address: P.O. Box 1348 

Charleston, WV 25325-1348 

Relationship to Document: Author 

(e.g., author of document, evaluation team leader, external auditor, internal auditor) 









